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ABSTEACT 

An impact evaluation of eight courts training pro 
(CTP) institutes funded by the Law Enforcement Assistance 
Administration was conducted. After a literature search and visit 
potential evaluation sites in all fifty states, twelve sites were 
selected from a random stratified sample of court systems. Data w 
obtained from 10M7 respondents (including trial and appellate jud 
prosecutors, defenders, and court administrators) via personal 
interviews and mail questionnaires. Findings included the followi 
(1) nearly 20 percent of the respondento cited training as a 
generator of organizational change within the criminal justice sy 
or personal change: (2) participants want training; (3) most send 
organizations support attendance at training programs in a variet 
ways; (t*) CTP institutes meet general training needs and provide 
setting in which individuals can exchange ideas and discuss probl 
and (5) training sessions suffer from shortcomings in needs 
assessment, balance of teaching methodologies, and instructor 
orientation-. Thirtyfour recommendations were made, and thirty-fi 
tables present detailed data summaries. (The case studies and sur 
instruments and a manual for conducting an on-going evaluation ar 
available through EEIC--see note.) (HN) 
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ABSTRACT 



This impact evaluation examines eight courts training institutes whfch h^ve been 
funded by the Lawj Enforcement Assistance Administration for a number of years. The 
Institutes provide training to trial and appellate judges, prosecutors, defi::^ders and 
court administrators. 

The objectives of the .♦udy were to: develop quantitative measures of training im- 
pact, measure the differential Impact of the various training Institutes and their pro- 
grams, determine the cost effectiveness of the individual programs in meeting their ob- 
jectives, and determine the Impact on the criminal justice system. 

The evaluation Inchrded personal interviews with training participants, and com- 
parison and peer-supervisor groups at 12 sites throughout the country. Mail question- 
naires were also used to survey some participants and a subsample of institute-related 
persons. Evaluators also made site visits to and attended training sessions given by each 
Institute. 

The final report sets forth findings and recommendations regarding each of the 
objectives discussed above. It also contains a "how-to" manual which describes the 
basic procedures for conducting an ongoing ev£iiuation of courts training programs. A 
bibliography on the subject of evaluation and criminal justice programs is also in- 
cluded. 
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SUMMARY 



PURPOSE 

Th9 Law Enforcement Assistance Administration, recognizing the Importance of 
enhancing the quality of criminal justice, has, through its sponsorship of the Courts 
Training Project (CTP), provided funds over a number of years to specialized training 
institutes to assure their continued operatiqn and utility. Pursuant to Its authority 
under ths Omnibus Crime Control and Safe Streets Act of 1968, LEAA contracted 
with McManis Associates, Inc. in October 1978 to conduct an impact evaluation of 
eight courts training institutes. These are: 



■ The National College of District Attorneys; 

■ The National College of Criminal Defense Lawyers and Public Defen- 
ders (now known as the National College of Criminal Defense); 

■ The Institute for Court Management; 

■ The National Judicial College; 

■ The American Academy of Judicial Education; 

• The Institute of Judicial Administration^ Appellate Judges Seminars; 
» The American Bar Association, Appellate Judges'" Conference; and 

■ The National Institute for Trial Advocacy, 

The objectives of the study were to: develop quantitative measures of training 
impact, measure the differential impact of the various training institutes and their 
programs, determine the cost-effectiveness of the Individual training projects in meet- 
ing their training objectives, and determine the Impact of the criminal justice system. 
In addition, the purpose of the evaluation was to make recommendations afrecting 
actions, decisions, and functions of LEAA and the training institutes. 



METHODOLOGY 

Although evaluations of some of the eight training institutes were performed in 
the past, for the most part, they did not attempt to measure training impact or to sys- 
tematically evaluate the training process. The first step In this evaluation process was to 
conduct a literature search of all pertinent information. The results of this research 
were useful in designing the data collection and analysis plan and identified some of 
the possible pitfalls involved with an evaluation of this type. 



The literature search assisted in refining the necessary data elements and sources. 
As the evaluation project evolved, two sets of subsamples were identified. These were 
institute-related subsamples and participant-related subsamples. 

The institute-related subsamples included core staff, instructors and members of 
the boards of directors. Preliminary data gathering included visits to the institutes to 
examine their management operations and programs, and to observe training sessions. 
The institutes provided names and mailing addresses for the institute-related subsam- 
ples. 

The participant-related subsamples included a participant group, a comparison 
group and a supervisor/peer group. Persons from each of these groups were either in« 
terviewed at each site or surveyed by mail. The final sample included a total of 1,047 
respondents. 

Also included in the study was a site subsample. This constituted 10 sites chosen 
from a stratified random sample of court systems which contained a high concentra- 
tion of CTP participants and two purposive sites chosen to fill in gaps which existed 
among factors considered important. 

Survey instruments for each subsample group were developed, and interview 
instruments were tested at a pilot site. The pilot test also provided the opportunity 
to train data collectors, to achieva a realistic idea of what data were obtainable, and to 
examine on-site procedures which had been developed in advance. 

To ensure a high level of cooperation among participants in the field, the method- 
ology provided for preliminary screening visits to each potential site prior to final se- 
lection. The visits involved meetings with the chief justice and/or presiding judge at the 
site, the court administrator(s), the chief prosecutor, the chief defender and the direc- 
tor of the state planning agency and/or regional planning unit. Each person was briefed 
on the purpose of the evaluation and participants' names and positions j^ere. verified 
Comparison and supervisor/peer groups were identified during the visif so that the total 
number of potential interviews at each site was known in advance. Following these 
visits, site selections were finalized, and visitation schedules were confirmed. 

Project teams visited a total of 12 sites over a three-month period. Interviews were 
conducted with trial and appellate judges, prosecutors, defenders, and court adminis- 
trative personnel. Following each site visit, data were analyzed to examine the impact 
of training on individual court systems. 

Another vital element of the methodology was the formation of a National Advi- 
sory Board. The Board was composed of eight nationally recognized experts in the 
criminal justice field who offered guidance to the evaluation team on all aspects of the 
project. Board members also reviewed a draft copy of the final report prior to submis- 
sion to LEAA. 
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FINDINGS 



Before summarizing the findings which resulted from the impact evaluation, some 
word? of caution are appropriate: 

• There are serious pitfails in atiampting to compare institutes in terms 
of relative impact. Each institute has its unique goals, objectives, 
curriculum, and participant group. Accordingly, the eight institutes are 
not truly comparable for the assessment of outcomes. 

■ With a number of respondents having attended more than one of the 
subject institutes, it would be hazardous to ascribe change to one insti* 
tutv at the exclusion of the other. Indeed, a few respondents had some 
difficulty distinguishing between training sponsors, training events, and 
outcomes involving more than one Courts Training Program. 

■ Field visits to the training institutes were conducted in late 1978. 
Findings resulting from thos^ visitations reflect conditions as they ex- 
isted then, unless otherwise specified. During 1979, the eight fining 
institutes had the opportunity to review the findings resulting from 
the visitations, and these were discussed with them by LEAA, Con- 
sequendy, a number of the weaknesses cited in the findings through- 
out this report have already been the subject of remedial action by 
the institutes. 

Overall findings of the study are presented below, with findings by individual in- 
stitute following. Although numerical data for each institute are not discussed in this 
summary, supplemental tables may be found in the main body of the text. 

Training Impact, Based on the findings of this evaluation, training is not the major 
impetus for either organizational change within the criminal justice system or personal 
change among those practitioners within the court system. Experience on the job and 
system changes caused by various circumstances emerge as the major causal factors of 
change. 

Nevertheless, the data indicate that training was cited by nearly 20 percent of the 
respondents in all role groups in the 12 court systems examined as a generator of per- 
sonal and/or organizational change. Therefore, training's contribution to the tota! 
climate which produces positive change cannot be dismissed. 

It is the view of the evaluators that measuring the impact of training per se is 
extremely difficult in light of the various constraints encountered and because there is 
a natural intertwining and cumulative impacting of such related factors as job experi- 
ence, education, training, advice of colleagues, reading, etc* Despite this, there are 
sufficient data available to conclude that training is a positive and important influence 
on the criminal justice system and on its practicing members. 
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paining Process. In examining the data gathered from those who have attended 
CTP institutes, several overall findings emerged: 

■ Participants want training. They beiieve in tiie principle of continuing 
education, and they feel that tfje legal system is ever-ciianging, thereby 
.•equiring them to increase their icnowledge, improve their skills, etc. 
They attend training voluntarily and enthusiastically. 

■ Afost sender organizations (e.g., courts, district attorneys offices, pub- 
lic defenders offices, etc.) strongly encourage the principle of continu- 
ing education and support attendance at training programs in a variety 
of ways (e.g., financial support, transfer of assignments, rearrangement 
of schedules, etc.). 

• The main strengths of the CTP institutes appear to be in meeting the 
general needs of the respective role groups and in providing a setting in 
which individuals can exchange ideas and discuss problems informally 
(often outside the classroom setting) with their peers from other states. 

Training, generally, may suffer from shortcomings in needs assessment 
balance of tenching methodologies (too much lecture), instructor orien- 
tation, and in attempts to meet individual needs. 

• There is an implied support for training programs which are about a 
week in length. The two-, three-, and four-week courses are often char- 
acterized as being too long (because of workloads, vacation conflicts 
and family commitments, concentration limits), and the two- and three- 
day workshops are often seen as being too brief 

■ CTP institutes need to pay more attention to the manner in which they 
mix trainees, especially as to , ole groups and levels of experience. 

Findings regarding individual institutes are presented below. 

INSTITUTE OF JUDICIAL ADMINISTRATION 

The Institute of Judicial Administration's Appellate Judges Seminars emerge from 
the data analysis as perhaps the best of the training programs in all respects except cost 
per participant. Based on participants' and instructors' responses, the Appellate Judges 
Seminars are widely recognized and respected. Participants gave high ratings regarding 
the relevance of the seminars to them and their courts, and data confirmed the general 
effectiveness of IJA s influence on personal and organizational change. 

The current needs assessment process appears to work well. The only potential 
weakness is the risk of having individual faculty members continually recommend that 
ttieir courses be repeated, without change, not necessarily because of "student" needs 
but because of the faculty member's familiarity with the course 
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One questionable area of the Appellate Judges Seminars exists with respect to 
travel and subsistence expenses which are paid to participating judges. Although many 
judges receive fund? from their jurisdictions, the cost differences are made up by IJA 
in the belief that without such assistance, judges would not attend the residential 
training programs. Since the seminars are important to the career development of 
appellate judges, and since many states provide reasonable salaries for appellate judges, 
the travel and subsistence funds might be better spent, unless there is a clearly demon- 
strated need. 

APPELLATE JUDGES' CONFERENCE (ABA) 

Participant and instructor perceptions of the Appellate Judges' Conference varied 
in several major areas, including program effectiveness and training design. Data show 
that a number of participants have no desire to return to future conferences. Another 
area of concern is that a relatively low percentage of the AJC participants have attemp- 
ted to make personal/professional changes upon returning from conferences while even 
fewer respondents attempted organizational changes after training. 

Instructors rate aspects of training design higher than participants, particularly 
regarding consideration of individuals' needs, clarity of learning objectives, degree to 
which objectives are demonstrable, opportunities to practice what is taught, and the 
degree of useful feedback received from instructors and peers. 

Participants gave the faculty high ratings in areas of substantive knowledge, practi- 
cal experience and teaching ability, but noted the scant representation by members of 
other than legal disciplines. 

NATIONAL JUDICIAL COLLEGE 

The data generated as a result of this evaluation indicate that the National Judicial 
College ranks high as an effective training institute, has a well-designed training pro- 
gram, and is effective in providing a mechanism for exchanges of views among peers. 
Instructors and participants alike rate the College high in effectiveness. Data indicate 
a high percentage of participants have attempted to make personal changes as a result 
of training, but a significantly lower percentage have attempted to make organizational 
changes. 

The NJC's training design is rated high by both participants and instructors. The 
College has an informal needs assessment process among its constituents, which may 
account for the relevant, welt-received programs. The only negative area identified was 
the degree to which the participant is kept informed of his/her progress throughout the 
course. 

The three strengths frequently mentioned by participants were: the collegiality 
generated among judges, the credentials and ability of the faculty, and the management 



of the training setting. The chief weaknesses cited were that some courses were too 
long (especially the four-week courses) and that lectures were used too extensively. 

AM ERICAN ACADEMY OF JUDICIAL EDUCATION 

Da^ collected during this evaluation indicate that AAJE is a reasonably effective 
training institute which is well-regarded by those who have attended its programs. Par- 
ticipants gave high ratings to training design, capability of faculty, and relevance to 
the needs of their court systems. Exceptional and widespread praise is given to AAJE's 
legal writing courses. 

Participants and faculty both expressed high satisfaction with the training settings 
at AAJE programs, but instructors were dissatisfied with the management of those 
training events. Specifically miintioned were inadequate orientation and training of 
faculty, inadequate adminirlrative prof:edures, high turnover among Academy staff, 
and the absence of a fu!!-cim' ^ijuucal director or leader at all programs. 

AAJE programs reflect' the desire to serve diverse needs of judges. The programs 
are designed for ease of replication, and enrollments to the national conferences are 
consistently high. At the present time, however, AAJE programs are scheduled based 
on past experiences of the Academy and judicial education trends, rather than on the 
basis of a systematic needs assessment. 

INSTITUTE FOR COURT MANAGEMENT 

ICM appears to be filling a national training void for court administrative person- 
nel. Among those court administrators and other Support staff who do attribute 
changes in their performance to the training they have experienced, ICM receives fre- 
quent mention. The relevance of ICM training generally received high ratings from both 
participants and instructors surveyed. On the other hand, more individuals were critical 
of ICM training programs than of any of the other institutes. 

According to the data, ICM is not effective, overall, in persuading participants to 
make changes in the way they perform their duties. Some participants are critical of 
the relevance of training programs and the match between trainees' needs and instruc- 
tor expertise. Instructors cite inadequate orientation and training, and insufficient ap- 
praisals of their performance. Many participants do not share training materials with 
others or recommend ICM. 

The Institute is currently reassessing its goals, objectives, and areas of future ex- 
ploration. The reassessment should prove useful to ICM in light of these findings. 

NATIONAL COLLEGE OF DISTRICT ATTORNEYS 

The data analysis showed that NCDA is a reasonably effective training institute. 
The College is effective in persuading participants to make changes in the way they per- 
form their duties. It is also an excellent mechanism for establishing cotlegial contacts 
and for instilling a pride of profession in prosecutors. During data collection, numerous 
suggestions by participants and instructors called for more cooperation between 
NCDA and the National College of Criminal Defense, even to the extent of conducting 
joint courses. 
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Participants gave faculty a high rating, but f-.. that lectures were relied on too 
heavily. Instructors expressed some dissatisfaction with NCDA, citing such problems 
as: inadequate orientation and training, inadequate program modification based on 
systematic assessments, and insufficient appraisai of performance. Both instructors 
and participants commented on NCDA's poor mixing of trainees by roles and experi- 
ences, overly large classes, and insufficient instructor contact outside the classroom, 

Th^: National College was the only CTP institute criticized for the insensitivity 
demonstrav^d by its staff and faculty at training programs with regard to females and 
minorities. Another serious complaint expressed by participants and observed by eval- 
uators was the laxity of attendance monitoring at NCDA regional workshops and the 
indiscriminate awarding of CLE credits. 

NATIONAL COLLEGE OF CRIMINAL DEFENSE 
(LAWYERS AND PUBLIC DEFENDERS) 

The National College of Criminal Defense also emerges from the data analysis as a 
relatively effective training institute. Instructors and participants alike feel training is 
relevant, and a high percentage of trainees adopt new methods and techniques learned 
at the College. NCCD was highly commended for reinforcing the role and confidence 
of the public defenders as a professional group. As in the case of NCDA, participants 
at NCCD recommended more integration between the two institutes, including joint 
programs. 

Faculty at NCCD received high ratings from participants. Instructors and Taculty 
alike gave the training setting high ratings, except for some dissatisfaction with the 
large class size. Instructors expressed concern over the management of training events, 
especially regarding orientation and training they are given. 

One concern which emerged during data collection (and expressed by previous 
evaluators) is whether the NCCD is pursuing its goal of improving the quality of repre- 
sentation for the indigent defendant, particularly with respect to private attorneys 
trained by the College. 

NATIONAL INSTITUTE FOR TRIAL ADVOCACY 

The National Institute for Trial Advocacy uses sound teaching techniques which 
are widely recognized, praised, and replicated. A high percentage of participants made 
personal changes on the basis of training, and most participants recommend NITA to 
others. 

NITA's faculty and teaching techniques (particularly role playing/simulation/ 
videotaping) received very high ratings by participants. Instructors and students both 
expressed criticism about the manner in which trainees are mixed regarding roles and 
experience. 

Although NITA's program management was rated highly, the evaluation team dis- 
covered deficiencies with respect to NITA's internal management. Such weaknesses as 



the absence of up-to-date files on faculty and trainees, and difficulty in obtaining cur- 
rent expenditure and budget information, indicated that tightening of management and 
administrative procedures would be beneficial. 



SUMMARY OF RECOMMENDATIONS 



As a result of the findings of this siudy, recommendations regarding individual 
institutes and the broader issues of national training programs funded by LEAA have 
been develop^ and are summarized below. 



IJA 

1. The Institute of Judicial Administration should amend its policy regarding the 
provision of travel and housing financial assistance and provide such aid only 
in cases of demonstrated financial need. 

2. In conjunction with IJA's adoption of the first recommendation, LEAA should 
further reduce its funding support in FY 1980 and should aim for complete 
withdrawal of financial support by FY 1982. 



AJC 

3. A more formal, systematic approach to assessing the needs of the audience and 
its satisfaction with the program would be beneficial to the program develop- 
ment process (of the Appellate Judges' Conference) and should be attempted. 

4. LEAA should increase its monitoring and ongoing evaluation of the Appellate 
Judges' Conference so that it can determine within the next year whether the 
program can be strengthened or whether funding should be terminated. 



NJC 

5. NJC should take appropriate and immediate action to ensure adequate minority 
representation on core staff. 



AAJE 

6. The Academy should reassess its goals and establish priorities for planning. 

7. The Academy should consider limiting its target audience to judges of limited 
jurisdiction, including those who are nonlaw-trained. 
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8. A formal needs assessment should be undertaken once the goals of the insti- 
tution have been reexamined. 

9. Efforts should be made to implement a faculty development program. 

10. The Judicial Education News, the bimonthly newsletter that was terminated for 
financial reasons three years ago, should be resurrected. 

11. The Academy catalogue, which contains policy and program information, should 
be rewritten for distribution, 

12. The National Videotape Library should be a priority concern of the Academy, 

13. The Academy should analyze the reasons for its relatively high staff turnover 
and take steps to correct the situation. 

14. The procedures manual should be revised, completed, and approved by the 
Executive Director as soon as possible to assure fairness and consistency in 
personnel matters. 



iCM 

15. The Institute should develop written policies and procedures for orienting instruc- 
tors to aid them in developing courses and presentations that are based on measur- 
able outcomes or objectives. 

16. The Institute should follow through on its plan to coordinate scheduling and pro- 
gramming of training events on court management with the National Judicial Col- 
lege. 

17. Both ICM and LEAA should participate in a reassessment of ICM to make de 
terminations concerning future directions, processes and funding levels for the 
Institute. 



NCDA 

18. The National College of District Attorneys should explore additional possibilities 
for sharing facilities, services, and personnel with the National College of Criminal 
Defense, including the possibility of joint training programs. 



19. NCDA should reorganize its core faculty and central staff to provide for a more 
realistic representation of women and minorities. 



20. NCDA should alter its training program monitoring procedures (and perhaps its 
monitoring personnel) to ensure that trainees are encouraged to attend sessions 
and that repeated absence will in some way be reported back to the sender organi- 
zation or at least not rewarded with the presentation of CLE credits. 



NCCD 

21. LEAA should closely monitor the future activities of the National College of 
Criminal Defense to ensure that progratn attendance is limited to attorneys pro- 
viding some reasonable amount of representation to indigent defendants. 

22. NCCD should move to develop a more systematic method for identifying the 
needs of the audience it serves. 

23. The College should reorganize its core faculty group to assure greater representa- 
tion of qualified women, blacks, and other minority groups. 

24. The College should take steps to assure staff representation of racial and ethnic 
minorities, in the spirit of equal employment opportunity. 

25. The College should assign the Dean and Associate Dean to assume responsibilities 
formerly assigned to the Director of Training for the preparation and training of 
faculty ipor all programs, and such preparation should be strengthened. 

26. LEAA should require the College to become more selective in the awarding of 
scholarships to private attorneys and insist on verification of the applicant's 
record of representing indigent defendants. 

27. NCCD and NCDA should formally develop plans for the increased sharing of 
costs, including those suggested in this report, to take fuller advantage of their 
physical proximity and related interests. 



NITA 

28. In view of the negative impact that a further reduction of funding would have on 
NITA's ability to grant scholarships to public attorneys engaged in criminal 
practice, LEAA should attempt to maintain its current level of funding support 
for such scholarships. 

29. NITA should unify the geographic location of its administration and establish a 
more efficient method of records management. 
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30. LEAA should develop an inventory of all courts training programs it funds, 
regardless of primery supporting division or office, for the purpose of coordi- 
nating the allocation of such funds. 

31. LEAA should consider ways to make SPA's and RPU's more aware of the impor- 
tance of courts training and to encourage increased local funding support, where 
warranted. 

32. LEAA should take steps, through budgetary action and program regulations, to 
require funded training institutes to share training materials and exchange ideas 
on effective teaching techniques. 

33. LEAA should discuss with the CTP institutes the wisdom of placing a greater 
emphasis on regional training programs as an answer to increasing travel costs 
tnd limitations on out-of-state travel* 

34. LEAA should move to increase its monitoring of CTP institutes by attending/ 
observing at least two training programs of each institute annually and by visiting 
each institute at least once a year. 
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CHAPTER I 
INTRODUCTION 



L INTRODUCTION 



A. BACKGROUND 

It has become evident over the last several years that the various participants in 
the American Judicial system-lawyers, judges, and court administrators-too often lack 
the skills and training required to carry out their responsibilities effectively. These de- 
ficiencies have hampered the ability of the courts to cope with the mounting pressures 
caused by high crime statistics and the increasing fine tuning of due Process. 

In assessing their responsiveness to changing needs, those involved in the criminal 
justice system have reexamined legal training methods. A key result has been to em- 
phasize more effective means of obtaining both efficiency and due process through 
continuing legal education. 

The Law Enforcement Assistance Administration (LEAA), recognizing the impor- 
tance of enhancing the quality of criminal justice, has, through its sponsorship of the 
Courts Training Project, provided funds over a number of years to specialized training 
institutions to assure their continued operation and utility. Eight national training 
programs have been selected by LEAA for evaluation to determine whether the proj- 
ects are meeting their objectives and whether they are impacting the judicial process in 
the United States: 

■ The National College of District A ttorneys (Houston) 

• The National College of Criminal Defense Lawyers and Public Defen- 
ders (Houston) (now known as National College of Criminal Defense) 

■ The Institute for Court Management (Denver) 

■ The National Judicial College (Reno) 

■ The American Academy of Judicial Education (Washington, D.CJ 

■ 77?© Institute of Judicial Administration, Appellate Judges Seminars 
(New Yorl<) 

■ The Appellate Judges' Conference of the ABA (Chicago) 

■ The National Institute for Trial Advocacy (Chapel Hill) 

Pursuant to its authority under the Omnibus Crime Control and Safe Streets Act 
of 1968, LEAA contracted with McManis Associates, Inc. in October 1978 to conduct 
an impact evaluation of these courts training programs. The objective of the evaluation 
were to: 

■ Develop quantitative measures of training impact; 



Measure the differential impact of the various training institutes and 
their programs; 



■ Determine the cost-effectiveness of the individual training projects in 
meeting their training objectives; 

■ Determine the impact of the training on the criminal justice system; 
and 

■ Afa/re recommendations regarding actions, decisions and functions of 
LEAA and the training institutes. 



LEAA intends to use the results of this evaluation to formulate future funding 
and project management/monitoring decisions. The training institutes will also be 
able to utilize the evaluation findings to learn from effective approaches used by other 
institutes and to identify weaknesses in their own training approaches. 



B. METHODS OF 

THE STUDY 

The methodology designed to conduct the impact evaluation included several key 
features: 

■ A literature search which enabled the study to interface with, and build 
on, other assessments of the Courts Training Project and the individual 
institutes. The search documented the methods and findings of other 
studies concerning impact evaluation of training for professionals, parti- 
cularly those which are court-related, 

" A sampling design plan which made it possible to capture data docu- 
menting the results of the various training efforts within each funded 
project and diffused over a number of court systems nationwide. 

■ The use of an advisory panel of court-related individuals to ensure the 
soundness and authenticity of the research design, to oversee the proto- 
col efforts and advise on relations with sampled jurisdictions, and to re- 
view and comment on the data analysis, 

RESEARCH DESIGN 

Literature Search 

Critical to the development of an effective evaluation is a thorough understanding 
of existing knowledge and data sources in the subject areas of courts training projects 
and the assessment of courts-related training programs. 

McManis utilized this fund of infomiation, conducting a search of literature 
applicable to the Courts Training Project and to the evaluation of training programs. 
Annual reports, prior evaluations, and project grant files were reviewed and analyzed in 
consultation with LEAA. At the same time, published documents pertaining to training 



program impact evaluations and the court systems were reviewed. Set criteria were ap- 
plied to this effort to ensure comprehensiveness of search. Among the criteria used were: 



Timeliness--was the material up to date? 

Direct relevance to this effort— i,e,, courts functions, performance mea- 
suras for court professionals, continuing education, evaluation of train* 
ing. 

Quality of publications, research methodology, etc., from which the 
information was obtained. 



The results of this literature search were used in refining the research and analysis de- 
sign and also in identifying similar projects for comparison to assess the relative cost 
effectiveness of the Courts Training Project. {See Appendix D.) 



Determing Data 
Needs 

Between the literature search and the construction of the data collection instru- 
ments falls the important task of identifying the data elements which will form the 
basis of analyses of trainee performance. The task of identifying data elements for this 
evaluation was viewed within a systematic construct of data generation utilization. In 
other words, identification of data elements was set within a data generation cycle 
which moves from the general objectives of the study to key questions, to speciftc data 
elements, and then to the kinds of measurements and analyses to be used to meet the 
study objectives. Thus, the basic data elements for the study were part of a data gen- 
eration cycle which might look like this: 



DATA GENERATION CYCLE 



Evaluation Objactivt 





Analysis 



Key Quastioni 





Data Me&suremants 



Data Elements 





Data Sources 



The data elements to be retrieved in the data collection phase of an evaluation 
must be integrally related to the specific study objectives and to the mode and thrust 
of the proposed data analysis. Failure to develop a systematic data generation plan is 
to court misdirection and to risk gathering too much, too little, or irrelevant data. For 
this reason it is well to define each segment of the data generation cycle: 

■ Evaluation Objectives. The specific intended evaluation outcomes. 

• Key Questiofis. The basic points of inquiry to be explored with respect 
to training projects and the various types of trainees which will enable 
the contractor to meet the evaluation objectives. 

• Data Elenients: The fundamental information items which must be 
gathered to answer the key evaluation questions according to perform- 
ance criteria, 

■ Data Sources. The one or several places where the data elements are to 
be found, along with the methodfs) for obtaining them, 

■ Data Measurements, The method of handling and arranging the gather- 
ed data in order to present them in forms that are ready for analysis. 

■ Data Analysis, The techniques to be used in examining the data to 
obtain findings which will satisfy the study objectives. 

In order to systematically obtain and analyze all the data necessary to achieve the 
evaluation objectives, data matrices were developed. They set forth the key questions, 
subquestions, variables, data elements, and data sources for each of the rr^ajor dimen- 
sions of the evaluation. {See Exhibit i.) 



Data Collection 

Instruments 

After the data needs and sources were identified, data collection instruments were 
designed. These included interview topic guides, obsen/ation checklists, and other in- 
struments which appear as samples in the Evaluation Manual (Appendix C). However 
the key set of data collection instruments utilized in this study were designed for on- 
site interviews with Courts Training Project participants, non-CTP comparisons, and 
third party or peers/supervisors; for mailing to CTP participants; for mailing to instruc- 
tors at ^e etght CTP institutes; and for mailing to the members of the governing 
boards of the eight institutes. These instruments appear \n Appendix B. 



OMB Clearance 



In conformance with existing requirements governing survey efforts which involve 
the collection of data from more than nine individuals or organizations, a clearance 
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KEY QUESTION: WHAT ARE THE RESULTS OF COURTS TRAINING? 



mmmm 

imiEnoiltQFCHAIISEI 



L TowfiitextmthmchingMin 
ind#iitt'MiKjgB.p«ftpec> 
tlve^mdvilu^ occurred K a ro* 
tultoltniniofl? 



VARMIlEt 



Knowledge 



■ Penpectin 



■ SellConcept 



2, To whit extent htvediingK in 
indhiiduils'role-relited under- 
standing mdtklii occurred k 
a result of training? 



Phllosopliy 



■ Values 



Technical Ability 



• Work Habits 



• Command of Role 



ColleglalltY 



OATAEL£MENT$ 



Changes In participants'* substantive knowledgi 



■ Changes In partidpanls' awareness of litues 
< ChangM In partklpaAtt' awareness of alternative IP- 
pfoaches to problems 



SOURCEI 



Paniclpant/Conipartsons 
Interviews Instructor 
Questioonalns 



Participant/Comparisons 
Interviews ISecondarv: Peer 
and Supervisor Interviews) 



Changes In participants' proleisional commltinent 
Changes in participants' caifidence as practitioners 



Philosophical changes in the way participants view their 



Reordering of role priorities 



> Changes In participants' technical proficiency in meet* 
Ingtaskrequirettranu 



Changes in participants' work habits, such as time man- 
agen^ent 



Changes In participants' capacity for understanding 
role^relBted problems 



Changes in participants' capacity for solving role-related 
problerns 



Changes In the amount of participants' communication 
and consultation with peers In other courts and juritdic- 
tions 



Participant/Comparisons 
Interviews 



Partlclpant/Cofliparisons 
Interviews 



Participant/Comparisons 
interviews ISecondary; Peer 
and Suparvitor Interviews) 



Participant/Comparisons, 
Peer, and Supervisor Inter- 
views; Instructor Question- 
naires 



P&rticlpant/Comparlsoos, 
Peer, and Supervisor In- 
terviews 



Participant/Comparisons 
Interviews {Secondary: 
Instructor Questionnaires! 



Participant/Comparisons, 
Peer, and Supervisor 
Interviews 



Participant/Comparisons 
and Peer Interviews 



MEASURES 



Open End 



ANAIVSEI 



Frequency, cross- 
tabulation with parti- 
cipants and comparl* 
son group, 
Cross-tabulation with 
role groups 



*Paf tidpant " hm having attended one or more ol the elj^t training institutes' sessions. 
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KEY QUESTION: WHAT ARE THE RESULTS OF COURTS TRAINING? - Continusd 



nMitEniOM 

lOIUEMIOUOFCHAXGEl 


VARIAILEt 


DATAELEII£I!TS 


SOURCES 


KEASURES 


ANALYSES 


1 Towhitixtrnthmchmgnln 
Individuals' KtlofflwdbdMV' 
Ion occumd IS 1 result of 
tninin^? 


e WorkAccomptithrwit 


■ Qiingos in the «a)ount of work pr(xiuc«d by participants 
i Chirniei In the quality of work produced by participants 


Participant/Comparisons 
andSupemlsor Interviews 


Oper 

\ 


lEnd 

/ 


Frequency, Cross- 
tabulation with par- 


■ Appllcitioiu 


« Changei In the number of technologleal, procedural, and 
other changes innltutad by participants in their courts 


farticlpent/Comparlsons, 
Peer, and Supenfisor Inter- 
views; Records and Re- 
ports; other studies 


songrouj 
Cross-tat 
rdegrou 

\ 


P. 

lolatbnwith 

/ 


1 IntiractloA 


1 ChinoH in lUftlrliuflti' ituU n( ^illnn utiiti thi mfi»ta\ 

* wMifyqi 111 pn uvipHiu wfw Ui yoiiiny wiin vnt pnvru 

public, court users, arvl Other Stall 

• Changos In others' responses to participants 


timcipanuwomparisonSf 
Peer, and Supervisor Inter- 
views; Other studies atxl 
observations 


4. TowhitixisnihivfiQQrtQito" 
dtiA9uoocumdisirituitol 

tnining? 


■ PmonnfiSubDIzition 


■ Rate of turnover among particlpanu in a given juris- 
diction 


Records 


\ 


Ratio 

/ 


> RelitlviPirtidpaot 
Iflflutnct 


< Amount of change Introduced by participants in rela- 
tion to size of jurisdiction 


Participant/Comparisons in- 
terviews jaggrtgatej; Records 


Cross-tabulation with 
jurisdiction 


■ InnovitivfltMtt 


> Amount of change Introduced by participants In rela- 
tion to particlpaAt ooocentritlon |hl{^-low| In a given 
jurisdiction 

> Rate of change per amount of training in a given juris* 
dialon 


Participant/Comparisons In* 
tenrlews jeggragatej; Records 


1 Cross-tabulation 
with concentra- 
tion level 

■ Cross-tabulation 
with amount of 
training 


■ ConsMiiui 


• Extent of agreement M the quality of justice pro- 
vkM in a given jurlidlction In relation to the con^ 
centratlon ihii^*low| of partidpanu 


Participant/Comparisons, 
Peer, end Supenrlsor Inter- 
views 


Ordiiul 


Cross-tabulation with 
concentration level 



**«llthlnthi|uriidictlon. ^ 
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KEY QUESTION: WHAT ARE THE STRENGTHS AND WEAKNESSESOF TRAINING? 



NIMOESntfiS 
MOtEttMBtUIOHl 


ViUiiuiEi 


DATAElEUin 


EOURCEl 


HEAHIREI 


mm 


1. HawrtifivmtistniningtQtht 
wodttttting? 


Apdicibjlitv to indfvid' 
uilntedi 


1^1^ ID wnioi ultninQ 11 ottigncd to N(KMS indlvKr 
ud's function md role 

i Dtgreetowtiichtriinlnglsdisigntdtoedleverete^^^ 


> Participant Interviews; In^ 

itructorOuHtionnairas; 
Curricula 

• Partidoanttilnstnjcton 


^k>minll 


Cro(s<t9bulationwlih 
groups of retpondentt 


Applicibllitytoiponsof 

logofumiatioo'irt* 

quirmnts 


■ l)egreetowlilditriinlf)gtirmiletnihewori(tttti^ 
• Degree to wtiichtriining it detigned to achieve trarater 
to the work setting 


■ Partidpentiilnsuucton 
e PertidoantsMiutnictan' 
Peers; Supervisors 


Nominil 


Frequency 
uosS'UDuiition with 
groups cl respondent! 


2. Towtutdagraeistrilning 
wctionad? 


Support of individual pir* 
ticipanu 


. • Degree to which individual deiiret to attend training 


Participants; Instructors 


Ordinal 


Frequency 


Support ol tender orgtfi- 
zition 


I Degree io which orgmizationencouragu attendance 
and reinlorcet/legitimim learning back honie 


Participants; Supervisors; 
Pters 


Nominal plus open 
end 


Frequency, summiry 


Support olprofealonil 
organlutioni 


■ Degree to which training it endoned by appropriate 
profettionalmenibenhipt 


Core stiff; Instructon; 

PartidpanttiPyoleisionii 

organizations 


Nominal 


Frequency 


3. Howuundiithotrfiinlngiih 

{KOlCh? 


Conaptuil Grounding 


> Degreetowhichtrainingdetignltba^oniindin^ln 
the theory of learning 

i Degreetowhichtniningitupgradedandfflodifiedin 
rdatjOA to current ttate<)f'the*art eleinenu of trainingi 


Instructor Questionnaires; 
Training materials 


Ordinal 


Frequency, 
Cross-tabulation with 
instructor type 


Methodology 


■ Degmtowhichtralningbbeudonneedsatteitment 

■ Degree to which training provides opportunltlet for 
pracdce 

■ Degreetowtiichtralningprovidetnrieaninghjl 
evaluation 

■ Degree to whldiadiievernent of objectives it oto 


Participant Interviews; 
InttiuctorQuettionnalret; 
Training meteriait 


Nominal 


Frequency 

Cross-tabulation with 
groups of respondents 


i Towhatdi^Nirfllriining 

contponenuretattd? 


fRnlS 


1 Degree to which participants' needi are expretted 
end considered 


Participants; Inttnictors; 
Application lomu;Evai' 
uatiom 


Ordinal 


Frequency 
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KEY QUESTION: WHAT ARE THE STRENGTHS AND WEAKNESSES OF TRAINING? - Continued 



SUt'QUESnOllS 
(PROGESOniEWOIlS 


VARIABLES 


DATA Ely Eim 


SOURCES 


MEASURES 


ANALYSES 


4. (ContinufidI 


Objectives 


• Degree to which expected outcomfl] ire presented to 
the learner 

• Degree to which expected ot)|ectivis ire dear arKltuc* 
cinct 

1 Degree to which expected learner it informed of progress 
re: objectives 


Participants; Instructors; 
Materials 


Nominal 


Cross-tabulation with 
type of respondent 




Activitifit 


■ Degree to which insuuctlonal events treit needs and 
objectives 


Participants; Instructors; 
Materials 


Nominal 


Summary 




Matftriils 


> Degree of continuitv between activities arid content 


Participants; Instructors; 
Materials 


Ordinal 


Cross'tabulationwith 
type of participant 


5. How"cr0dibli''isifioin]truc* 
tlonil iutfinlh«8yesofthe 
pirtlciptntt?* 


hrcalved Competence 


> Degree to which instructors' substantive, practical, and 
teaching expartise Is valued by participants 


pBrticlpanb^Postsesilon 
evaluations 


Nominal plus open 
end 


Crossiabuiationwith 
type ol instructor, 
summary 






1 Degree to which instructors are called upcn for continuing 
technical assistance after training 


Instructors; Core SUff; 
participants 


Nominal 


Frequency 






< Number of times that instructor is invited back to 
teach/train 


Instructors; Core Suff 


intemal 


Mean 


6. Ittutittitttdnaitcociwyedind 
dMmted by tiw programs? 


Continuing ralitions 
wi(tilnstitution(sl 


1 Desire for additional training 

« Desire for InstltutlDm/instructort to provide follow-up 

tedmical assistance 
> Frequency of return to training 


Partidpints; Instructors; 
Core Staff (Outreadi Data) 


Nominal 
Interval 


Frequency 
Mean 




Goilrefenncing 


■ Degree to which partidpints say thty "buy into" goals 

■ Degree to which participants attempt to malte goal' 
related behavioral and systemic changes 


Participants 


Open end 


Summary 



*Panidpant " Parson liaving attended sessions of one or more of the light training Institutions, 



KEY QUESTION: WHAT ARE THE STRENGTHS AND WEAKNESSES OF TRAINING? - Continued 



SUMUESDOn 
(PROCESS DIMENStONSi 


VARiAILES 


OATAELEKEKTS 


SOURCES 


MEASURES 


ANALYSES 


6. IConMI 


Multipliar Effects 


" Degree to which participant! publicize Irutitutions to others 
■ Degree to which partidpanU and instructors use institution 
mdterials outside 


Partidpanti; Peers; Su- 
pervisors 

Participants; Initructors 


Nominal 


Frequency 


7. It the ssttitig conducive to leim- 
ing? 


Composition of popula- 
tion 


■ Representativeness (of race, tex, age, geographyl; homo- 
geniety (of functions, experiences! 


Participanu; instructors; 
Core Staff; Prerequisites 


Nominal 


Frequency 


ProgrBm Structure 


> Class size; duration; location 


Participants; instructors; 
Core Staff 


Interval; Nominal 


Mean, Frequency 


Staffing Pattern 


■ Match between instructor expertise and parlicipant 
roles/needs; instructor availability; number of contact 
hours 


Participants; Instructors; 
Core Staff; Evaluations 


Nominal: Interval 


Frequency, Mean 


Modaiing 


< Degree to which desired behaviors and skills are rein- 
lorcea in various aspens oi iraining 


Particlpantt; Instructors; 
Evaluationi 


Nominal 


Frequency 


Facilities 


« Adequacy of support services 


Participants; Instructors; 
Evaluations 


Ordinal 


Frequency 


8. By whet process in the 
training evfint$inin808d? 


Needs Anilysit 


> Degree to which programs are based on training population'i 

DrflfKiiniul nfwwk 

pivlHHlUIWI IKnJUi 


Core Staff; Instructors; 
Participants 


Nomina 
end 

\ 


plus open 

/ 


Frequen 
mary 

\ 


:yand Sum^ 

/ 


Direction; Communication; 
Cocrdinatbn 


■ Degree to which objectives, procedures, and responslbll' 
ities are clear 


Instructors 


Problem-Solving 


■ Degee to which procet^res exist and are utilized 


instructors 


Feedt)ad and training for 
Inttructofs 


■ Degree to which instructors are appraised of expecutions and 
performance; given orientation and developmental training 


Instructors 


Evaluation 


■ Degree to which program modification is information* 
based 


Instructors; Participanu; 
Core Staff 
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KEY QUESTION: WHAT ARE THE STRENGTHS AND WEAKNESSES OF TRAINING? - Continued 



tuuuanon 

(TROKiSDIIIEIflQeiS 


VARUUEt 


DATAElEltEim 


tOURCEl 


UEASURES 


AKALYSEI 


9. BywhitprGOttmir9C0(eit<ft 
optfttiomnunigfid? 


Dtosion Making 


■ Degree to which dedslont are pdicy'faated 


Board Minutes; Board 
Member Questionnaire 


Nominal] 
end 

\ 


)(usopen 

/ 


Frequer 
Summi 

\ 


Kyand 

ry 

/ 


Communicition 


1 Deivatowhichdecitioniaretrantrnittedto 
organtiitloonwnbert 


Cora Staff 


CoordinitloA 


1 Oegnt to which atithority is delegited and lines ol 
autotty are dear 


Core Suff; Job Desalp' 
tions 


i>robl6m'SolvinD 


1 Degree to which proi)leniioh!ng procedures are 
definKlandutiiized 


Core Staff; Procedures 
Manuals 


Monitoring 


■ Degree to which policy changes are influenced by 
leedbacl( and needs attettrmnt 


Board MInuUs; Board 
Member Questionnaire; 
CoreSuff 


10. IMiit Art th« unit |l.e.. par* 
ticjpmt-diylcomMod' 
Atidwithdifhnflttvpeiol 
tnining? 


■ Coitt 

i Training Inpun 
1 Pirtidpanti 


■ Scope, nature, and duration of each seminar 

■ Total costs covered t»y Institution funds 

■ Total nurnbefolpartidpanti 


■ Budgets 
1 Grant applications 
> Annual reports 
" Core stiff interviews 


Open End Ratio 


Summary 

DesaiptiveStatit' 

tics 
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package was prepared and submitted to the Office of Management and Budget in 
February 1979 and approved approximately two months later (O.M.B. No. 043- 
S790006). 

TRAINING INSTITUTES 

The eight training institutes identified previously were, of couree, the focal points 
of this evaluation. Field activities to collect necessary data fell into two general phases. 
One consisted of visits to the training institutes for on-site examination of xheir man- 
agement, operations and programs, plus observation of their training programs in ses- 
sion. The second phase consisted of data collection at 12 randomly selected court sites 
where training participants, their supervisors, and comparison subjects were interview- 
ed and the impacts of training were examined. 

Visits to the training institutes served several purposes: 

a Training officials received an explanation of the impact evaluation 
strategy and had an opportunity to comment on and contribute to the 
study design; 

» Detailed information was obtained concerning the type of training pro- 
vided, dates and locations of training events, curriculum, rosters of 
participants^ eta; 

■ fi/fanagement information was collected concerning each institute's 
operations, including staff and faculty qualifications, funding and fiscal 
administration, etc.; 

■ Key institute personnel were interviewed concerning goals, objectives, 
training needs and targets, training methods, etc.; and 

■ The physical facilities of tho training institutes were observed, as well 
as their equipment and support services. 

Data collection instruments and guides which were prepared for these visits ap- 
pear as samples in the Evaluation Manual in Appendix C. 



Training Data 

Collectors 

All members of the evaluation team received training in the use of data collection 
instruments and topic guides prior to field visits to the training institutes. The first in- 
stitute visited, the American Academy of Judicial Education, was used as a training 
site for all of the evaluators because of its convenient location. 



Visits to 

Institutes 

All eight of the training institutes were visited by teams of two or three evaluators 
(except AAJE, which was visited by eight evaluators) within the first three months of 
the project. Each visitation was headed by a senior member of the evaluation team, and 
evaluators spent an average of three person-days at each institute. In all cases, the 
necessary data were collected during the visit and/or subsequently made available by 
the institute. Full cooperation was received in all instances. 



Verification of 
Findings 

Following the visitations, reports were drafted reflecting the findings of the evalu- 
ation teams. These draft reports were then made available to the institute directors or 
deans so that they had the opportunity to correct factual errors or misunderstandings, 
and so that they could add explanatory information where appropriate. 

The findings resulting from the institute visitations appear in Chapter II. 



Observations of 

Training Programs 

One training program conducted by each of the training institutes was observed 
by the evaluators. An observation guide and rating instrument (see Evaluation Manual, 
Appendix C) were developed and utilized to assure uniformity of coverage. 

Since each training observation consisted of an average of only two person-days 
and involved only one of several training[ programs offered by each institute, they were 
not used as mrasures of training efficiency and effectiveness. Rather, they provided the 
evaluators with an experience of a typical training event of each institute and a sample 
of the management and conduct of such programs. They also afforded evaluators an 
opportunity to speak with training participants and instructors during a training 
experience. 

SAMPLE DESIGN 

The overall sampling strategy was to survey participants in the eight training insti- 
tutions and those responsible for setting the institution's policies and presenting the 
training sessions. 

The sample for the study was divided into two groups of subsamples: 

■ Institution-related subsamples 

■ Participant-related subsamples 
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The institution-related subsamples are the training Institution's instructors, core 
staff, and members of the board of directors. Instructors who have taught training 
sessions during the past three years were included in the instructor sample and survey- 
ed by mail. In addition, the appropriate core staff of each institution were surveyed. 

All members of the boards of directors of the eight training institutions were also 
surveyed by mail. 

The participant*related subsamples are: 

■ Site sample; 

■ Participant sample; 

■ Supervisor/peer sample; and 

■ Comparison group sample. 

The site sample was selected from the list of 60 sites having a high concentration 
of former participants in the training institutions. The sample frame was developed 
from the training institutions' records on residence of participants. Sites were included 
if they had at least one participant from each of the eight training institutions. If no 
sites within a state met that criteria, the site(s) with the most institutions represented 
was included. Therefore, all 50 states had at least one site in the sample frame. This 
procedure was used to assure national representation and adequate representation of 
each of the eight training institutions while still limiting the number of on-site visits. 

A stratified random sample was used to select 10 of the 12 sites to be included in 
the site sample. The stratification variable was Federal region* One site for each of the 
10 Federal regions was randomly selected. This procedure assured'r^presentation of all 
areas in the nation in the sample. 

Some factors which were considered important for the site sample, were: 

■ Concentration of CTP participants; 

■ Variability in type of courts in the site; 

■ Urban/rural representation; 

» Presence of a unified court system; 

• Presence of automated court systems, evidenced through use of 
PPlOMIS;and 

■ Representation of non-tawyer fudges. 

After the stratified site sample was selected, two additional sites were selected to fill 
gaps that existed among the factors considered important. These sites were chosen 
randomly from the group of similar sites in the original sample frame. (See Exhibit 2.) 
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CHARACTERISTICS OF SITE SAMPLE 



Structure: 


Unified Court System 


Nonunif ied Cc 


Geograhical 
Area 


Rural/Small 
Population 


Urban/Large 
Population 


Rural/Small 
Population 




f Participants: 




Site No. 2 
Site No. 8 






Concentration o1 


Site No. 4 
Site No. 11 


Site No. 1 2 


Site No. 10 
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Within each sample site, all participants of training institutions were included in 
the survey sample. This sample included trial judges, intermediate appellate judges, 
state supreme court justices, non-lawyer judges, prosecutors, defenders, court adminis- 
trators, court clerks, and others. A participant subsample was selected for on-site inter- 
views. The remaining participants were surveyed by mail if ft was not possible to con- 
duct interviews with them. 

A second interview sample for each site included participants' supervisors and/or 
peers. 

A third sample was interviewed at each site. The comparison group sample con- 
sisted of individuals who were matched to the participants in the participant interview 
sample on the basis of job role* The comparison group, however, had not attended any 
of the institutions' training sessions. 

in summary, the final sample included: 

481 training institute participants; 
210 comparisons; 

74 supervisors/peers; 
225 instructors; and 

57 members of institute governing boards. 

1,047 

Information concerning participants, comparisons, instructors and governing 
board members is displayed in Tables 1-6. 

Information detailing those in the study group at each of the 12 sites is shown in 
Tables 7A'7L. 



Screening Visits 

One technique which contributed to the success of the sample selection and the 
fact that the numbers of individuals in each category approached the maximum targets 
was the screening visit made to each potential jurisdiction before final site selection 
was determined. These screening visits, conducted by senior evaluation staff, usually 
involved meetings with the chief justice of the state, the presiding judge at the site, the 
state court administrator, the chief prosecutor, and the chief defender, as well as the 
director of the state planning agency or regional planning unit. The purposes of the 
screening visits included verifying the presence of training participants whose names 
were supplied by the institutes, selection of comparison and supervisor interviewees, 
collection of necessary data, and assessment of the degree of cooperation and support 
the site visit would be accorded. 



TABLE 1 
NUMBER IN STUDY 



Role Group 


CTP Participants* 


Comparison 


Total 


NCDA 


NCCD 


ICM 








AJU 


Nil A 


Total 


Appellate judge 








5 

w 


8 


14 








24 


73 


Trial judge 


1 




1 




00 








127 


66 


193 


Private attorney 


6 


29 




1 




— 


— 


11 


47 


6 


53 


Prosecutor 


85 


2 












9 


96 


60 


156 


Public Defender 


2 


69 












2 


73 


28 


101 


Court administrator 






27 


5 










32 


14 


46 


Court clerk 






12 












12 


5 


17 


Other 


14 


4 


21 


2 


1 






3 


45 


7 


52 


Total 


108 


104 


61 


115 


32 


14 


22 


25 


481 


210 


691 



Three appellate judgei, six trial judges, two private attorneys, one public defender, and two court administrators were counted once for each of two institutes for which a 
training assessment was completed. 
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TABLE 2 

CHARACTERISTICS OF CTP PARTICIPANTS 





Sex 


Racial/Ethni 


c Background 


Highest Degree 


Type/Avg. Years Ex 


perienco 


noie uioup 




F 


Asian 


Hispanic 


Black, 
not 

Hispanic 


White 


H.S. 


Bachelor's 


Master's 


Law 
Degree 


Law/ 
Court 


Professional 


Currmt 
Position 


Appellate judge 


45 


1 








46 








46 


29.3 


30.2 


4.6 


Trial judge 


112 


9 


1 




6 


114 


4 


2 




115 


23.7 


24.3 


6.7 


Private attorney 


38 


7 






1 


44 








45 


7.3 


8.3 


4.3 


Prosecutor 


88 


8 


2 


2 


4 


88 






1 


95 


7,3 


8.1 


4.0 


Public defender 


64 


8 


2 


2 


4 


64 






1 


71 

f 1 


5.6 


6.7 


3.1 


Court administrator 


25 


5 








30 


6 


9 


8 


7 


10.6 


15.0 


4.6 


Court clerk 


8 


4 








12 


7 


3 




2 


18.3 


23.3 


7.4 


Other 


33 


12 






3 


45 


8 


13 


9 


15 


11.0 


13.2 


4.2 


Total 


413 


54 


5 


4 


18 


443 


25 


27 


19 


396 
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TABLE 3 

CHARACTERISTICS OF CTP COMPARISON GROUP 





S« 


X 


Racial/Ethni 


cBackpund 


Highest Degree 


TYpe/Avg.YeaREx 


perience 


Role Group 


M 


F 


Ameri- 
can 
Indian 


Hispanic 


Black, 
not 

Hispanic 


White 


H.S. 


Bachelor's 


Master's 


Ph.D. 


Law 


Law/ 
Court 


Professional 


ifUrrBiii 
Position 


Appellate judge 


23 


1 


1 




1 


22 








1 


23 


33.4 


34.3 


8.0 


Trial judge 


62 


4 




1 


7 


58 










66 


25.4 


26,0 


8.6 


Private attorney 


6 










6 










6 


10.2 


10.8 


9.8 


Prosecutor 


4fi 


12 




2 


2 


56 










58* 

WW 


1 

Q. 1 


Di/ 


3.0 


Public defender 


19 


9 








28 










28 


4.4 


5.0 


2.7 


Court administrator 


7 


7 








14 


6 


3 


4 




1 


9.1 


14.2 


2.4 


Court riprk 


0 
L 


Q 
0 








5 


4 








1 


8.8 


17.6 


6.6 


Other 


7 


0 






1 


6 




2 


1 


2 


2 


13.4 


17.0 


4.0 


Total 


174 


36 


1 


3 


11 


195 


10 


5 


5 


3 


185 









*Two respondents did not provide highest degree. 



40 
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TABLE 4 
AMOUNT OF TRAINING 





CTP Participant! 


Comparisons 


Role Group 


Average 
Number of Courses 
Last 5 Years 


Average 
Total 
Number of Courses 


Average 
Number of Courses 
Last 5 Years 


Average 
Total 
Number of Courses 


Appellate judge 




Q Q 


I.I 


1 7 

It/ 


Trial judge 


3.3 


4.1 


2.6 


2.9 


Private attorney 


3.3 


3.4 


4.0 


4.4 


Prosecutor 


3.0 


3.3 


2.8 


2.8 


Public defender 


3.9 


4.1 


2.9 


3.0 


Court administrator 


3.8 


4.0 


0.8 


1.1 


Court clerk 


3.8 


4.1 


0.8 


0.8 


Other 


2.0 


2.5 


2.0 


2.1 
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TABLE 4A 
TYPE OF TRAINING TAKEN 





CTP Participants 


Comparisons 


Role Group 


Average 

No. 

CTP 
Courses 


Average 
No. 
State 
Courses 


Average 
Na Other 
Federal 
Counes 


Average 
No. 
Private 
Courses 


Average 
No. 
Other 
Courses 


Average* 

No. 

CTP 
Courses 


Average 

No. 

State 
Courses 


Average 
Na Other 
Federal 
Courses 


Average 
No. 
Private 
Courses 


Average 
No. 
Other 
Courses 


Appellate judge 


2.3 


0.9 


0.1 


0.8 


<0.05 


06 


13 


02 


u.o 


(19 


Trial judge 


2.1 


1.6 


.0.1 


0.3 


0.1 


02 


24 


01 






Private attorney 


1.8 


0.6 


<0.05 


1.0 


0.1 




1.0 




2.7 


1.5 


Prosecutor 


1.5 


0.5 


0.3 


0.7 


0.2 


<0.05 


1.3 


0.2 


1.2 


0.2 


nil* If 1 

Public defender 




0.6 


0.1 


0.9 


0.1 


0.1 


1.6 


<0.05 


1.2 


<0.05 


Court administrator 


2.1 


0.8 


0.4 


0.7 


0.3 




0.1 


0.2 


0.2 


0.6 


Court clerk 


2.3 


0.4 


0.3 


0.6 


0.4 




0.4 




0.4 




Other 


1.4 


0.3 


0.3 


0.3 


0.1 


0.3 


1.1 




0.6 


0.1 



•OTP panidpanu mn troitid bs comparison] if the inititute course m taken more thin 10 years ago or if they felt they could not tamemLer enough to completa the training 
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TABLES 



CHARACTERISTICS OF CTP INSTRUCTORS 







ChiiutiiisticsolCTFIiistructon 


HighistDigrit 


AvinH Vun 

niwiB^ 1 Mil 

LiN/Court 


AvinQi Viin 
P(ofeaionil 


AnngiYtin 






S«x 


Nitiva 




1 






No 




w 










No 


Auaciittd 
Nithlnstituti 


Instituti 


N 


M 


F 


Aincricin 


Aiiin 


BiKk 


HispiDic 


Whiti 


RnpoRtt 


HS 


BS 


Mistin 


PhD 




Midicil 


Rtspoiua 


Expifitnct 


Expiriince 


iilnstniclor 


NCOA 


44 


40 

191%) 


4 

19%) 


1 


1 


- 


- 


41 


1 


1 


3 


3 


2 


30 


6 


- 


13.4 


16,7 


2,8 


NCCO 


25 


24 

196%) 


1 

14%) 


- 


- 


- 


1 


23 


1 


- 


1 


- 


- 


20 


2 


2 


16,2 ■ 


17,2 


3,6 


ICM 


28 


26 

193%) 


2 

17X1 


1 


1 


- 


1 


24 


1 


1 


6 


8 


7 


7 


- 


- 


11.2 


17,1 


4,0 


NJC 


51 


47 

m 


4 

18%) 


- 


- 




- 


51 


- 


1 


1 


- 


4 


45 


- 


- 


22,3 


23,9 


5,3 


AAJE 


28 


26 

193%) 


2 

im 










28 






1 


1 


5 


21 






17,1 


23,0 


6,1 


IJA 


7 


6 

186X1 


1 

(14%l 


1 








6 












7 


* 




17.7 


20.7 


6,7 


AJCIABAI 


21 


21 

liooxi 












19 


2 


1 








20 






19,1 


27.9 


4.0 


NITA 


21 


16 

|76%| 


6 

I24X) 






1 


1 


18 


1 










21 






12.9 


13,4 


4.2 


Total 


225 


206 
192%) 


19 
18%) 


3 


2 


1 


3 


210 


6 


4 


11 


12 


18 


171 




2 


16,2 

a 


10.9 


4,2 
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TABLE 6 

CHARACTERISTICS OF CTP BOARDS OF DIRECTORS 



Institute 


Number of 
Board Members* 


Number of 
Respondents 


Rate 


HVBrags Tears 
on Board 


Id ot Kespondents Wno 
Attended Institute Training 


% of Respondents 
Affiliated with Other 
Judicial Training Programs 


NCDA 


13 


6 


46% 


8 


50 


100 


NCCO 


12 


8 


67% 


4 


75 


88 


ICM 


11 


8 


73% 


5 


100 


50 


NJC 


15 


8 


53% 


5 


75 


75 


AAJE 


7 


5 


71% 


5 


100 


100 


1 lA 


11 




15% 


4 




50 


AJCIABA) 


12 


8 


67% 


4 


88 


88 


NITA 


17 


10 


59% 


7 


80 


70 


Total 


117 


57 


49% 









'From lists supplied by each Institute. 

"Two respondent! did not fill out questionnaires but sent explanatory letters. They ere not included in calculailons. 



TABLE 7A (Sits 1) 
NUMBER IN STUDY 



Role Group 


CTF Participants* 


Comparison 


Supervisor/ 
Third Party 




NCDA 


NCCD 


iCM 


NJC 


AAJE 


IJA 


AJC 


NITA 


Total 




Appellate judge 












1 






1 




2 


Trial judge 










1 








in 




3 


Private attorney 


4 


2 












1 


.7 






Prosecutor 


13 


1 












1 


15 


2 


2 


Public Defender 




6 














6 


3 


1 


Court administrator 






















2 


Court clerk 






4 












4 




2 


Other 




1 


6 












7 




2 


Total 


17 


10 


10 


9 


1 


1 




2 


50 


5 


14 



^Om private attorney counted once for each of \m institutes. 



TABLE 7BlSite 21 
NUMBER IN STUDY 



Role Group 








OTP 


Particip 


ants 






Comparison 


Supervisor/ 
Third Party 




NCDA 


NCCD 


ICM 


NJC 


AAJE 


IJA 


AJC 


NITA 


Total 




Appellate judge 














1 




1 


1 




Trial judge 








3 










3 


4 


2 


Private attorney 




3 














3 






Prosecutor 


8 
















8 


8 


2 


Public Defender 




2 














2 






Court administrator 
























Court clerk 






1 












1 




1 


Other 


1 


1 














2 






Total 


9 


6 


1 


3 






1 




20 


13 


5 



TABLE 7C (Site 3) 
NUMBER IN STUDY 



Ra|0 nrnuD 


CTP Participants* 


pAmn^ricnn 
l/UinpailoUil 


Supervisor/ 
Third Party 


NCDA 


NCCD 


ICM 


NJC 


AAJE 


IJA 


AJC 


NITA 


Total 


Appellate judge 






1 






1 


3 




4 


1 




Trial judge 








9 


3 








12 


19 


1 


Private attorney 




1 














1 






Prosecutor 


9 
















9 


2 


1 


Public Defender 




















2 




Court administrator 






1 


1 










2 




1 


Court clerk 






1 












1 






Other 


1 




3 












4 






Total 


10 


1 


5 


10 


3 


1 


3 




33 


24 


3 



'One court administrator was counted for each of two Institutes. 



TABLE 7D (Site 4) 
NUMBER IN STUDY 



noi6 uroup 


CTP Participants* 


Comparison 


SiinftniitAry ^ 

Third Party 


NCDA 


NCCD 


ICM 


NJC 


AAJE 


IJA 


AJC 


NITA 


Total 


Appellate judge 










1 


1 


3 




5 


1 

1 


1 
1 


Trial judge 


1 






1 










2 


2 


1 
1 


Private attorney 
























Prosecutor 


3 
















3 




1 


Public Defender 


1 


14 












1 


16 


5 




Court fidtninictratnr 

WMI % UUIIIIIIIJllUlV* 






11 


1 










12 


9 


1 


Court clerk 






2 












2 


1 




Other 


1 




1 


1 


1 








4 


2 


1 


Total 


6 


14 


14 


3 


2 


1 


3 


1 


44 


24 


5 



}ne appellate judge was counted once for each of two institutes. 



TABLE 7E (Site 5) 
NUMBER IN STUDY 



Role Group 


CTP Participants* 


Comparison 


Supervisor/ 
Third Party 


NCDA 


NCCD 


ICM 


NJC 


AAJE 


IJA 


AJC 


NITA 


Total 


Appellate judge 










1 


2 


1 




4 






Trial judge 








11 


2 








13 


8 


2 


Private attorney 




3 












1 


4 






Prosecutor 


12 
















12 


12 


2 


Public Defender 




9 














9 


1 


1 


Court administrator 






2 


1 










3 






Court clerk 






1 












1 






Other 




2 


2 












4 




1 


Total 


12 


14 


5 


12 


3 


2 


1 


1 


50 


21 


6 



•One appellate judge, one private attorney, and one court administrator were counted once for each of two institutei. 
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TABLE 7F (Site 6) 
NUMBER IN STUDY 



Role Group 


CTP Participants* 


Comparison 


SuDfirvitnr/ 

Third Party 


NCDA 


NCCD 


(CM 


NJC 


AAJE 


IJA 


AJC 


NITA 


Total 


Appellate judge 










2 


1 






3 


1 

1 




Trial judge 








14 


6 








20 






Private attorney 




3 












2 


5 






Prosecutor 


2 
















0 




1 


Public Defender 
























Court administrator 






2 












2 






Court clerk 
























Other 


1 
















1 






Total 


3 


3 


2 


14 


8 


1 




2 


33 


7 


1 



Two trial judges have been counted once for each of two institutes. 
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TABLE 7G (Site 7) 
NUMBER IN STUDY 



Role Group 


CTP Participants 


Comparison 


Supervisor/ 
Third Party 


NCDA 


NCCO 


ICM 


NJC 


AAJE 


IJA 


AJC 


NiTA 


Total 


Appellate judge 








1 


1 




5 




7 


3 




Trial judge 








14 


1 








15 


11 


2 


Private attorney 




6 




1 








2 


9 


1 




Pmeo/>|itnr 
riUdCitUiui 


1 

1 














2 


3 


■5 
0 


1 
1 


Public Defender 




7 












1 


8 


4 


1 


Court administrator 






1 


1 










2 


1 


1 


Court clerk 






1 












1 






Other 
















1 


1 






Total 


1 


13 


2 


17 


2 




5 


6 


46 


23 


5 



TABLE 7H (Site 8) 
NUMBER IN STUDY 



Role Group 


OTP Participants* 


Comparison 


Supervisor/ 
Third Party 




NCDA 


NCCD 


ICM 


NJC 


AAJE 


IJA 


AJC 


NITA 


Total 




Mppeiiaie juage 














3 




3 


4 




iridi juQy^ 








16 


2 








18 


2 


1 


rrivaie aiiorney 


1 


2 












3 


6 


1 




Prosecutor 


IJ 
















13 


3 


2 


Public Defender 


1 


10 














11 


1 


1 


Court administrator 






5 


1 










6 


2 


1 


Court clerk 






1 












1 


3 


1 


Other 


5 




1 












6 


3 


1 


Total 


20 


12 


7 


17 


2 




3 


3 


64 


19 


7 



One trial judge and one public defender wore counted once for each of two institutes. 
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TABLE 71 (Sites) 
NUMBER IN STUDY 



Role Group 


CTP Participants 


Comparison 


Supervisor/ 
Third Party 




NCDA 


NCCD 


ICM 


NJC 


AAJE 


IJA 


AJC 


NITA 


Total 




Annollato iiiri<iD 
nppclldlc jUUvJc 












2 






2 


5 


3 


Trial liiHriA 








1 










1 


4 




Priusto ntff\rna\i 

rfivdic duorncy 




1 












2 


3 






Prosecutor 


n 
3 














1 


10 




2 


Public Defender 




11 














11 




1 


Court administrator 






















2 


Court clerk 






1 












1 






Other 


1 




4 










1 


6 






Total 


10 


12 


5 


1 




2 




4 


34 


9 


8 
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TABLE 7J (Site 10) 
NUMBER IN STUDY 



Rolfi (irmin 

llVIv \Jl uuu 








tir 


Participants 






Comparison 


Supervisor/ 
Third Party 




NCDA 


NCCD 


ICM 


NJC 


AAJE 


IJA 


AJC 


NITA 


Total 




Appellate judge 








1 


1 


3 


5 




10 


4 


2 


Trial judge 








9 










9 




1 


Private attorney 
























Prosecutor 


5 
















5 






Public Defender 




1 














1 






Court administrator 






1 












1 
1 




1 


Court clerk 
























Other 






4 


1 










5 






Total 


5 


1 


5 


11 


1 


3 


5 




31 


4 . 


4 



TABLE 7K (Site 11) 
NUMBER IN STUDY 



Role Group 


CTP Participants* 


Comparison 


Supervisor/ 
Third Party 


NCDA 


NCCD 


ICM 


NJC 


AAJE 


IJA 


AJC 


NITA 


Total 


Appellate judge 








2 




2 






4 




1 


Trial judge 








10 


5 








15 


2 


2 


Private attorney 


1 


4 














5 






Prosecutor 


n 

0 














4 


10 


9 


2 


Public Defender 




4 














4 


6 


3 


Court administrator 
























Court clerk 


















• 


1 




Other 


3 
















3 






Total 


10 


8 




12 


5 


2 




4 


41 


18 


8 



appellate judoe and three trial judges were counted once for each of two inititutei. 
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TABLE 7L (Site 12) 
NUMBER IN STUDY 



Role Group 








CTP 


Participants 








Comparison 


Supervisor/ 
Third Party 




NCDA 


Ncrn 


IPM 


M IP 


AAJE 


IJA 


A IP 


NITA 


Total 




Annpll;)t0 iiirino 
r^^pwiidic juuijc 








1 


2 


1 


1 




r 

0 


4 


1 


Trial iitrlr*o 
1 1 idi |uu{|.(; 






i 

1 


c 

b 


3 








9 


n 

8 




riivdic diiurncy 




A 
4 














4 


4 




Prosecutor 


4 


1 












1 


6 


17 


1 


Public Defender 




5 














5 


6 


2 


Court administrator 






4 












4 


2 


2 


Court clerk 
























Other 


1 














1 


2 


2 


2 


Total 


5 


10 


5 


6 


5 


1 


1 


2 


35 
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DATA COLLECTION FOR IMPACT EVALUATION 



Six data collection instruments were designed for use in this evaluation. They are: 

■ Training participant questionnaire (mail survey) 

■ Training participant interview 

■ Comparison group interview 
• Supervisor/peer interview 

■ Board of Director's questionnaire (mail survey) 

■ Instructor questionnaire (mail survey) 

(An example of each of these instruments appears in Appendix B.) 

^lembers of the data collection/field survey teams were trained rn the use of the 
interview instruments and in other matters related to the site visits and data collection 
generally. A procedures manual {see Appendix C) was developed for training purposes 
^nd for ongoing use of the data collectors during the course of the project. 



Pilot Test 

In order to test the interview instruments (and in some cases the mail question- 
naires) and further train the data collectors under actual field conditions, a pilot 
test was conducted in March 1979 in a relatively small, urban, unified court system not 
far from Washington, D.C. The pilot test also permitted the evaluators to test all 
aspects of managing site visits and to examine the workability of the plans and proce- 
dures which had been devised in advance. 

As a result of the pilot test, changes were made in the procedures manual, and 
some adjustments were made in data collection instruments. 



Site Visits 

Following the selection of the 12 sample sites and the screening visits, discussed 
previously, site visit schedules were formulated and arrangements made, including the 
setting of interview appointments with training institute participants, comparison in- 
dividuals, supervisors, and other appropriate personnel at each of the sites. 

The field surveys began in late May and were concluded in late August 1979. 

Each of the 12 sample sites was treated as an individual case study, and a report 
was prepared describing each jurisdiction within the site, the types of interview sub- 
jects who were contacted, and the findings at the site. The 12 case studies appear in 
Appendix A* 



C. ROLE OF THE 

NATIONAL ADVISORY 
BOARD 

To augment the capabilities of the evaluation project team, a National Advisory 
Board was established, composed of nationally recognized experts in the field of 
criminal justice admit Jstration. The eight members of the Board were: 

Dr. Junius Af/ison, Professor Emeritus, Vanderbilt University Schoo/ of Law; 

Dr. Ralph C. Bledsoe, Professor, Federal Executive Institute, U.S. Office of 
Personnel Management; 

Ms. Frances Boronski, Chief of Civil Court Services, Administrative Office of 
the Courts, State of New Jersey; 

Dr. Charles Edelstein, Assistant Professor, Department of Criminal Justice 
Administration and Law, Florida Atlantic University; (recently appointed 
County Judge, Dade County, Florida); 

James Farrar, Court Administrator, 61st District Court, Grand Rapids, 
Michigan; 

Ms. Nancy Goldberg, President, National Defender Institute, Chicago, 
Illinois; 

Marshall Hartrnan, National Defender Institute, former National Director of 
Defender Services for the National Legal Aid and Defender Association; and 

Honorable Edward Pringle, Former Chief Justice, Colorado Supreme Court; 
(currently Director of Legal Research and Writing Program, University of 
Denver College of Law). 

The National Advisory Board fulfilled an extremely important set of functions. 
The Board did not merely have a review function to perform but played an integral 
part in the way the project evolved. 

The mix of the Board was vital. Members were specifically chosen because their 
education, experience, and professional orientation covered the fields of judicial 
administration, court administration, legal education and trial advocacv. which were to 
be the dominant concerns of the study. 

The National Advisory Board members had the capacity to offer methodological 
guidance to the research team* They could help in formulating and refining the re- 
search design, particularly as it related to performance indicators, to the sample design 
and the size of the population which was to be studied. They were helpful in their sug- 
gestions of corroborating data that could be obtained from the training sites where a 
process evaluation was to be undertaken and at the courts where the experimental 
group and controls were to be interviewed. 
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From the background of their experience the Board could offer practical insights 
into the operation of the system and to guide the evaluators. Some Board members 
were currently working in courts: an appellate judge, court administrators, researchers, 
practicing attorneys and educators. It was felt that the Board members could help to 
open the system to the research team through their insights. They also advised the di- 
rector of the project about the proper protocol to be followed in setting up a research 
team at a court site for a few days in order to draw from the site the desired informa- 
tion. 

Certain members of the Board, because of their involvement in a great number of 
cognate projects, or who, themselves, had been intimately involved in training pro- 
grams of the type to be evaluated, were looked upon as ongoing resource persons, to be 
called when questions arose concerning methodological, practical, or other problems. 
As indicated above, the Board members were representative of the different role types 
under study, and they provided an orientation to their unique points of view. As re- 
searchers, the evaluation team had an excellent theoretical background in the work 
that the role types performed, but the Board members had an "insider's" point of 
view. 

The Board net at three critical points in the life of the study. At these meetings 
they assisted in ihe evaluation of the project itself * critically evaluated the "do^ability" 
of the project, evaluated instruments for data collection and aggregation, and re- 
viewed interim reports which had been sent to them. At the meetings they discussed 
research strategies. They helped the evaluation team to clarify the evaluation criteria 
and were instrumental in focusing on key issues. On the threshold of the site visits, 
they worked with the staff in the evaluation of the pilot study project. Revisions were 
made in the instruments after a discussion with the staff. They helped in the develop- 
ment of a site typo; gy, making possible a better selection of sites. 

Finally, the products of the study (this final report) were distritiuted to the Board 
and reviewed by them prior to submission to the LEAA. 



D. CONSTRAINTS 
Methodoiogy 

The major methodological constraints facing this impact evaluation resulted from 
the fact that the evaluation took place long after the start of the training programs, in 
some cases over a decade later. Because of the ex post facto nature of the study, it 
was not possible to randomize the selection of the experimental and comparison 
observations, nor was it possible to make pre- and post-attendance analyses in order 
to assess the impact of the training. 
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Impact Measures 



The unavailability or unreliability of "objective" performance data within the 
local criminal justice systems led the evaluation team to rely on participant/com- 
parison self-reports of incidence of change as the impact measure. In addition, the wide 
differences among the institutes with respect to clientele, history, objectives, method- 
ologies, training formats, funding sources, and other resources made systematic com- 
parisons among Institutes extremely difficult. Thus, the quantification of impact find- 
ings in this report is intended to provide general indications of institute performance 
based upon respondent perceptions of applying what was learned. 



The Comparison 
Group 

The comparability of the comparison groups was limited by several factors: some 
comparison respondents were new hires or new appointees who had not yet attended 
training; others considered themselves too "old" or too busy to attend training. Thus, 
while the average numlDer of years of experience may be comparable, the comparison 
group tacks genuine comparability. Still other comparsion respondents had attended 
training programs not included in this evaluation, making them "participants" in other 
court-related training. Where this condition prevailed, the study obviously did not have 
a strict quasi-experimental situation of treatment nontreatment. 



Analysis 

While the site sample provided a good representation of court systems and the 
case studies provided a "flavor" for systemic impact, the resulting participant sample 
was too small to meaningfully disaggregate the findings by the duration of the training 
event attended, number of years since training, and the number of years' experience in 
one's job. This was a major constraint in determining the differential effects of various 
types of training provided by a given institute. To the extent possible, the impact sec- 
tion of the case studies takes note of these variables. 



Quantifiable 

Confirmatory Data 

A more general constraint involved the lack of quantifiable supporting data. This 
was true to a certain extent for all sites and all roie groups. Little data were available 
from the sites themselves; background material was frequently supplied by studies 
made by the National Center for State Courts. Information obtained from State Planning 
Agencies (SPA's) and Regional Planning Units (RPU's) was negligible. Local court ad- 
ministrators had little to offer beyond statistical summaries that they supplied to the 
State Court Administrators. Some court administrators simply referred the evaluators 
directly to the state court administration. The annual reports of the state court admin- 
istrators provided little data that could be used to support statements of change or 
impact made by participants or controls. Administrative officials in prosecuting and de- 
fending agencies rarely had data to offer, except of a very general nature, even in those 
states where PROM IS was being used. As the National Advisory Board had pointed 
out, statistical data across sites are not uniform, and meaningful comparisons could not 
be made. 
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The capturing of corroborative data, if and where available, would have required a 
much more intensive collection effort than was feasible given the time and cost con- 
straints. The number of sites would have had to have been curtail^, and a team would 
have had to remain on site for several weeks, establishing rapport with professionals, 
administrators and clerks, and aggregating the data in the files. 



Data Collection 

it was assumed that interviews with supervisors could yield some measure of 
changed performance on a pre- and post-training basis. However, supervisors' informa- 
tion, especially that of presiding judges, tended to exhibit only limited knowledge con- 
cerning their colleagues' actual job performance. When the interviews were not 'Void 
for vagueness," it seemed almost a point of honor for presiding judges not to know 
what goes on in the courts of other judges, even subordinates. Thus, it became diffi- 
cult, if not impossible, to ascertain whether training programs were producing impact 
on judicial demeanor, sentencing, speed, administration, etc., at least from the informa- 
tion supplied by presiding judges. This problem was not as great among prosecutors, 
defenders, or court administrators who were in a better position to observe the per- 
formance of junior or support staff. 



Locating and 

Classifying Participants 

Another problem encountered by the evaluators in securing the data was the dif- 
ficulty in locating members of the experimental group. This was partly caused by the 
rapid turnover of personnel. In addition, some of the lists furnished by the training in- 
stitutes were inaccurate. 



Respondent Recall 

A somewhat more serious constraint resulted from the inability of members of 
the participant (experimental) group to be able to clearly separate what had been de- 
rived from the program being evaluated as distinct from other programs that had been 
attended. This seemed to be particularly true of the assistant district attorney group, 
since members had attended a variety of in-state or in-service programs. Some had at- 
tended special out-of-state programs in organized crime but had also received training 
from NCDA on the same topic. This inability to differentiate between programs tended 
to obscure somewhat the validity of their observations concerning the impact of the 
Houston program. Respondents who were unable to recall specific training events 
were treated as comparison interviews in the analysis. 



The Scope of 

the Study 

Although respondents sometimes mentioned areas of unmet needs (which are 
noted in the report), it was not an objective of this evaluation to perform a needs 
assessment or a systematic assessment of unmet needs. Likewise, it was not within the 
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scope of the study to determine the extent to which the institutes wece reaching their 
respective constituent groups, i.e., penetrating the universe of potential trainees on a 
nationwide basis. The case studies do, however, give a general indication of the amount 
of training which has occurred within the local jurisdiction. 

In summary, a training evaluation of this type is susceptible to a number of 
natural constraints-*some endemic to ex post facto studies and others unique to the 
work environment of the Court Training Programs clientele. During the study design 
the following major constraints emerged: 

■ Quaf titative measures of training impact were only available through 
participants' self-reports of change with respect to knowledge, proce- 
dures, and priorities. 

■ The determination of differential impact and cost effectiveness of the 
respective institutes was hampered by the dissimilarity of the institu- 
tes' programs, missions, and clientele. 

■ The determination of impact of the training upon the criminal Justice 
system also had to be based upon participant/peer perceptions, set 
within the contextual framework of the local case study. 

Thus, while the evaluation team was able to pursue and address the original study 
objectives, the methodology and findings were limited by the aforementioned con- 
straints. Accordingly, as long as these constraints are understood, the study findings 
should provide the respective institutes and LEAA with insights for improving or af- 
firming the policies and approaches of the Courts Training Program. 
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CHAPTER II 
FINDINGS 



II. FINDINGS 



This chapter sets forth the findings of the study as they apply to the eight training 
institutes, their training programs, and the impact of that training as measured by 
various types of changes which have or have not occurred as a result. 



A. THE TR AINING 

INSTITUTES AND 

TRAINING PROGRAMS 

Eight training institutes which participate in LEAA's Courts Training Program are 
the central focus of this evaluation. They are: 

■ The Institute of Judicial Administration, provider of the Appellate 
Judges Seminars; 

■ The American Bar Association 's Appellate Judges ' Conference; 

■ The National Judical College; 

9 The American Academy of Judicial Education; 

■ The Institute for Court /Management; 

■ The National College of District Attorneys; 

■ The National College of Criminal Defense Lawyers and Public Defen- 
ders (now known as the Nationai College of Criminal Defense); and 

■ The National Institute for Trial Advocacy. 

Data which serve as the basis for these findings were collected during evaluation 
team visits to each of the training institutes, which included interviews with key insti- 
tute personnel, examination of pertinent records and documents, and observation of 
ongoing activities; during the on-site ob^rvation of selected training programs con- 
ducted by the institutes; from the mail questionnaire responses of faculty and members 
of the governing boards of each of the institutes; and from interviews with training 
institute participants in 12 randomly selected court systems throughout the country. 

Data collection efforts were initiated by obtaining and analyzing a variety of 
written material concerning each institution and its training programs. This was fol- 
lowed by site visits to each of the eight training institutions. The site visits were under- 
taken to: 

• Familiarize each institution with the purposes of t^ impact evaluation; 
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Observe the institution's center of operations and conduct interviews 
with key officials and staff; 



Obtain information and documents pertaining to the training programs, 
including course objectives, curriculum design, faculty and participant 
rosters, and dates and locations of courses; 

Obtain information concerning the management of the training institu- 
tion and its programs, including staff qualifications and responsibilities, 
funding sources and patterns, and fiscal management; and 

Seek the assistance of training program directors in subsequent data col- 
lection activities, at training programs selected for observation and in 
selected court systems throughout the country. 



Methodology 

Several preparatory steps were taken prior to site visits to assure the maximum 
use of staff time at each of the eight training institutions. Interview instruments were 
designed and tested to obtain management, program and funding information. Letters 
were sent to each institution explaining the format of the planned visit, suggesting a 
schedule, and listing the categories of information which could be assembled before- 
hand for discussion during the visitation. 

To familiarize themselves with the institution to be visited, each interview team 
reviewed literature and reports regarding the institution and its programs well in ad- 
vance of the visit. These materials included grant applications, progress reports, pre- 
vious evaluations, brochures and training materials. 

Each site visit was conducted by at least two evaluators and ted by a senior mem- 
ber of the project staff. The length of time required for each visitation varied, but the 
average was four to five person-days and two to two-and-a-half calendar days. The 
typical agenda included a kick-off meeting with the executive director and key staff of 
the institution to explain the purpose of the visit, followed by several individual inter- 
views with appropriate staff, utilizing the survey instruments as a point of departure. 
An exit interview, usually with the executive or institution director, was scheduled at 
the conclusion of the visit to review the data collection effort. At only one of the eight 
institutions were the evaluators able to observe training programs in progress during the 
site visit. Training programs of other institutions were monitored at later dates. 



Cautions 

Some words of caution are appropriate here, as examination of the eight training 
institutes begins: 

■ There are serious pitfalls in attempting to compare institutes in terms 
of relative impact Each institute has its unique goals, objectives, cur- 
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riculum, and participant group. Accordingly, the eight institutes are not 
truly comparable for the assessment of outcomes. 

With a number of respondents having attended more than one of the 
subject institutes, it would be hazardous to ascribe change to one insti- 
tute at the exclusion of the other. Indeed, a few respondents had some 
difficulty distinguishing between training sponsors, training events, and 
outcomes involving more than one Courts Training Program. 

Care should be taken in interpretation of the percentages used in the 
various tables when the number of respondents is small. 

Finally, with respect to those findings developed during the institution 
site visits, the visits were completed in late 1978 and the findings reflect 
the conditions that existed at the time, unless otherwise specified. Dur- 
ing 1979, the institutes and LEAAseparately and jointly— reviewed 
our findings and preliminary recommendations. A number of the weak- 
nesses reflected in the findings have already been the subject of 
remedial action by the institutes. 
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1, INSTITUTE OF JUDICIAL ADMINISTRATION 
APPELLATE JUDGES SEMINARS 



The Institute of Judicial Administration, New York University School of Law, 
which is responsible for conducting the Appellate Judges Seminars, was visited on 
November 30, 1978 by a two-member interview team. Prior to the visit, extensive tele- 
phone interviews had been conducted with two former directors of the Appellate 
Judges Seminar Program. The site visit was initiated with a meeting with the office 
manager to review documents which had been assembled for the evaluation team, the 
proposed schedule for the visit, and the IJA personnel who were to be interviewed. The 
team then met with the new director of IJA to explain the purposes of the impact 
evaluation and the site visit. A lengthy and detailed interview was then conducted with 
the director, utilizing the project director quesionnaire and other appropriate interview 
instruments as guides. Subsequent interviews were conducted with the office manager 
and with the fiscal officer, and numerous documents were collected and reviewed with 
them. Because of o^er pressing business in the office (the preparation of a state grant 
application), it was not possible for the evaluation team to conduct an exit interview 
with the director, but a brief status report was made to the office manager. In addition, 
a draft of the findings contained in this report was submitted to IJA for confirmation 
purposes. 



History of Institution 

The Appellate Judges Seminar Program of the Institute of Judicial Administration 
was the first major program developed for the continuing education of appellate court 
judges. Initiated in 1956, the seminars have been attended by approximately half the 
Supreme Court and intermediate Courts of Appeal judges in the country, including at 
least three members of the U,S. Supreme Court. The program format has been relative- 
ly unchanged for the past 22 years. Two residential seminars are offered each summer, 
one for judges of the highest appellate courts and one for judges of the intermediate 
appellate courts. Each seminar lasts approximately two weeks, and the number of par- 
ticipants at each is limited to about 20 judges. New appellate judges are given prefer- 
ence in the selection process. 

The Appellate Judges Seminar Program was founded and nurtured by Professor 
Robert A, Leflar during his 20-year term as its director. He continues to sen^e as con- 
sulting director. Professor Leftar was succeeded as director by Professor James C. 
Kirby, who held the position for two years, until 1978. The present director is 
Nicholas Scoppetta, who has an extensive background of public service in New York 
City. He was appointed Director of IJA in July 1978. 

The staff of IJA numbers nine, but only the director and the office manager are 
considered the core staff for the seminar program. Two other IJA staff members pro- 
vide a minimal amount of administrative support time to the seminar program. 
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The policy governing the Appellate Judges Seminars is established jointly by the 
Dean of the New York University School of Law and the Director of the IJA. 

The Appellate Judges Seminars have been funded in part by LEAA since 1973, 
but considerable financial support has been provided by other sources through IJA. 
The most consistent source of support has been West Publishing Company, which has 
provided an annual $25,000 grant to the seminar program since 1973. The pattern 
of LEAA funding has been as follows: 

1973 - $51,000 

1974- $44,605 

1975 - $45,000 

1976 - $45,000 
1977 -$63,299 
1978 -$39,950 



Goats and Objectives 

of the Institution 

The goals and objectives of the Appellate Judges Seminar Program are to provide 
continuing judicial education and to improve the quality of decision making and the 
administration of the appellate courts. 

These words are not explicitly stated in offical documents but were gleaned from 
various written material and from interviews with key seminar officials. 



Programs 

Trends: During the past five years (actually, during the past 22 years), the pro- 
gram goals of the Appellate Judges Seminar Program have remained constant— to up- 
date the knowledge of appellate court judges as to procedures, techniques and issues. 
There has been no deviation from the general program content and format of the 
seminars. 

Current: The programs presented at the most recent Appellate Judges Seminars 
were as follows: 

INTERMEDIATE APPELLATE JUDGES SEMINAR 

Judicial Administration including Supervisory Administration, the Decision- 
al Process, Relationships between Intermediate and Higher Appellate 
Courts, Efficient Administration of an Entire Judicial System, and 
Technological Aids. 

Appellate Review in Criminal Cases. 
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New Developments in Conflict of Laws. 

Appellate Review of Decisons of Administrative Agencies. 

Principles and Techniques of Statutory Interpretation. 

Free Press and Fair Trial. 

Law and the Computers. 

Nature and Function of the Appellate Judicial Process. 

Opinions including Critique of Quality of Judicial Opinions, Style in Judicial 
Writing, and What a Law Editor Looks for in Judicial Opinions. 

State Courts and the Federal System. 

Current Trends in Accident Law. 

SENIOR APPELLATE JUDGES SEMINAR 

All of the programs listed for the Intermediate Appellate Judges Seminar^ 
plus the following: 

U.S. Supreme Court Current Decisions. 

These programs were presented at two separate residential seminars, each consist- 
ing of two five-day weeks of sessions. A typical day extended from 9 a.m. to 4 p.m 
and consisted of three sessions of M/2 hours each. The faculty for each seminar con- 
sists of approximately 12 members including experienced judges, law professors and 
administrators. Most subjects are taught on a team basis, and participation and discus- 
sion by all faculty and seminar attendees are strongly encouraged. 

Needs Analysis and Curriculum Development: A cyclical curriculum planning 
process has been established over the years which combines the elements of the needs 
analysis and curriculum development. Simply stated, the process includes the collection 
of information from faculty and participants after each seminar. The participants, 
through informal conversations with seminar leaders and through anonymous evalua- 
tion questionnaires, comment on the strengths and weaknesses of the seminar and the 
subjects that were (or were not) presented. The faculty, in a more formal manner, meet 
with the program director to discuss the seminar and program content, the participant 
responses, and their impressions of the participants. {The latter subject is addressed for 
the purpose of identifying possible future faculty members.) 

As a result of these actions, the program director is able to develop a preliminary 
list of topics to be presented at the next seminars. About three months prior to those 
seminars, the preliminary list of topics is sent to the registrants for their review and 
comments in order to determine if the needs of the participants are being addressed. 
Based on the responses, the seminar program is shaped into final form. The process of 
faculty and participant input is repeated at the end of the seminar. 
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Faculty Selection and Development: As stated previously, the faculty of the 
Appellate Judges Seminar Program consists of approximately 12 members including 
experienced judges, law professors and administrators. Virtually all of the faculty are 
"graduates" of the Appellate Judges Seminars who were identified as potential faculty 
because of their performances at the seminars and/or because of the national reputa- 
tions they earned subsequently. The composition of this core faculty is fairly constant. 
Changes are made only when participant evaluations indicate that the faculty member 
is weak in presenting topics or when the faculty member chooses to step down. In ad- 
dition to this core faculty, specialists are utilized from time to time, including the 
editor of a publishing company which is a financial supporter of the seminars, and 
NYU Law School faculty. Of the 12 core faculty members (as of the time of our visita- 
tion) one is female and none is a member of a racial or ethnic minority group. The 
average age of the faculty is 55, with ages ranging from 36 to 71. All of the faculty 
members are or have been members of the judiciary. The average tenure of the faculty 
in the Appellate Judges Seminars is five years. Each faculty member is paid $1 10 a day 
and must be available for each day and each session of each seminar- 
While each faculty member is responsible for preparing his/her own course ma- 
terial, it must be submitted to the program director approximately three months in 
advance of the seminars for the director's review and approval. 

Permanent Staff: Although the staff of the Institute of Judicial Administration 
totals nine full-time personnel, only two are listed as core staff for the Appellate 
Judges Seminars. They are the director and the office manager. The director is said to 
spend approximately 25 percent of his time on matters pertaining to the Appellate 
Judges Seminars and is funded by the LEAA grant for 25 percent of his annual IJA 
salary. The office manager says she expends about 30 percent of her time on seminar 
program matters and her annual salary is supported by the LEAA grant to the extent 
of 30 percent. In addition, a secretary spends about ten percent of her time annually 
on seminar administrative support, and the fiscal officer spends about 20 percent of his 
time annually on seminar-related activities. Neither of these two staff members is sup- 
ported by the LEAA grant. 



Effectiveness 

Marketing of Programs and Selection of Trainees: The Appellate Judges Seminar 
Program has, over the years, enjoyed such recognition and acclaim as to minimize 
the need for extensive marketing. There is an annual mailing to each appellate court 
judge in the nation announcing the senior or intermediate Appellate Judges Seminar 
and inviting applications. (Invitations are sent to judges who attended a previous inter- 
mediate seminar but who have since moved to a senior appellate court.) In nearly every 
year except 1978, the number of applicants exceeded the desired limit of 20 partici- 
pants per seminar, so a screening process takes place in which participants are selected 
on the basis of certain criteria. These include a preference for new appellate judges, a 
limit of only one judge from each jurisdiction, and a preference for younger rather 
than wider judges who might be approaching retirement. In addition to the response to 
the annual mailing, word-of-mouth promotion by judges who participated previously 
produces applicants. (It is not unusual for an appellate court to register a newly ap- 
pointed judge immediately, based on the experience and recommendations of his/her 
new associates.) 



11-7 




Monitoring and Assessment of Programs, Faculty and Trainees: The monitoring 
and assessment of programs and faculty are carried out in two ways-through observa- 
tions and informal discussions during the course of the seminars, and through more 
formal assessment procedures at the conclusion of seminars. The informal techniques 
involve the program director ?.id the consulting director, who observsand participate 
in the seminars, speak with participants as to their impressions and reactions, and meet 
with faculty members to discuss seminar progress. At the conclusion of the seminars, 
participants are requested to fill out an anonymous questionnaire in which they assess 
the seminar program and faculty. At the same time, the faculty formally assesses the 
program content and the participants, the former for the purpose of recommending 
program changes and the latter to identify potential faculty for the future. 

Outside Evaluations: The Appellate Judges Seminar Program was evaluated in 
1975 by Stanley J. Heginbotham. Appellate judges who were surveyed commented 
favorably on the value of the seminars and attributed changes in their professional per- 
formance to things they had learned through the program. Most highly valued were re- 
views of recent U.S. Supreme Court decisions and the opinion writing course. The one 
negative finding growing out of the evaluation was that curriculum planning tended to 
be fixed with one individual, the program director. As a result of that observation, the 
planning process was broadened to include greater input from the faculty. 



Other Institution 
Activities 



Publications: Several books have been published as a result of the Appellate 
Judges Seminar Program. They include: 

• R. Leflar, Appellate Judicial Opinions, West, 1 974. 

R. Le*".?r, Ime.ynediate Operating Procedures in Appellate Courts 
American Bar Foundation, 1976. Ht^t^'^^ie i^ourts. 

■ R . A I d ise rt. The Judicial Process, West, 1976. 

External Relations: As noted previously, the Appellate Judges Seminars have re- 
ceived some financial support since 1973 from a publishing company, which also parti- 
cipates in the seminars. In rddition,. there exists an "informal understanding" wim 
the American Bar Associatic- concerning the Appellate Judges Seminar Series of the 
Appellate Judges Conference of the ABA in Chicago. The "understanding" is that the 
IJA program wilt focus on new appellate judges whereas the ABA pro£ im will provide 
continuing education to more experienced appellate judges. As for relations with 
LEAA, some literature of IJA recognizes LEAA as among its many sponsors However 
the brochure devoted exclusively to the 1979 Appellate Judges Seminars contains no 
reference to LEAA. 

Alumni Development: Close and warm relations are maintained with the alumn! 
of the Appellate Judges Seminars. They are not solicited for financial or other sup- 
port, but the "graduates" have been consistently supportive of the seminar program 
m recommending it to their associates. The alumni reception that IJA sponsors at the 
ABA annual meeting is wf il jttended (250 in 1978). 
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Program and 

Related Costs 

Budget History: The comparison of recent budgets is somewhat difficult foi 
following reasons: 

■ The 1976 program year fLEAA grant) extended from March h 1976 to 
February 28, 1977, whereas the budget for 1977 covered an additional 
month (March h 1977 to March 31, 19771 

m At the time ofourvisit, the IJA fiscal off icer could not provide us with 
a budget status report for the current fiscal year. 



Despite those factors, the following comparisons can be made: 



Revenue 


1976 
(3/1/76-2/28/77) 


1977 
(3/1/77-3/31/78> 


1978 
Budget Only 
(4/1/78-3/31/79) 


Tuition & Fees 
LEAA Grant 


$45,000 


$61,249* 


$39,950 


Publishing Co. Annual 
Contrib. 


25,000 


25,000 


25.000 


From Publishing Co. 
Reserve 


10,297 


7,678 


rj,742" 


Total 


$80,297 


$93,927 


$90,692 

1 



*An additional $2,050 in unexpended fund: was returned to LEAA. 
"From Publishing Co. Reserves and/or from other reserves depending on degree of need to meet 



travel and subsistence costs of participants. 



As the above figures indicate, total program costs over the past three years have 
rfcT-.ained fairly constant. Despite fluctuations in LEAA funding levels, the annual con- 
tribution from a publishing company and the availability of contribution reserves have 
made it possible for IJA tn maintain total program spending at the levels indicated. 

One of the major expense items is the travel and housing costs for participants. 
This amounted to $26,565 in 1976 and $30,917 in 1977. That cost was estimated at 
$27 192 for 1978 but finai figures were not available at the time of our visitation. 
Seminar participants are urged to seek travel and subsistence funds from their State 
Planning Agencies and/or from their courts training programs. However, if those 
sources cannot cover travel and maintenance costs, seminar officials believe they must 
provide the funds in order to encourage appellate judges to attend the residential 
training programs. LEAA funds are not used for these purposes. The travel and living 
expenses are provided from IJA's matching money. Fortunately, the 
tion frorn the publishing comp3.ny, the contribution reserves, (and some $400,0(X} 
which IJA has in time certificates), make such matching possible. 



11-9 



Cost Analysis: There are a number of ways in which instructional costs may 
be displayed, and each is subject to challenge because of the difficulty in assigning dol- 
lar costs to the many facets of an Instructional program of this nature. However, in 
order to provide some measure of the costs involved, we offer two methods of calcula- 
tion. 

a. In 1977, a total of 43 appellate judges attended the two seminars. They at- 
tended a total of about 50 hours of classroom instruction during the two week periods 
of the seminars. Since the total cost of the seminar programs for 1977 amounted to 
$93,928 (including travel and living expenses), it might be stated that the cost of in- 
struction was $43.50 per participant per instructional hour. 

b. Also in 1977, the total of 43 appellate judges attended ten days of instruc- 
tional seminars. Since the total cost of the seminars was $93,928, it can be stated 
that the cost of instruction was $218 per judge per instructional day. 

Management 

The Institute of Judicial Administration and the Appellate Judges Seminar Pro- 
gram appear to be managed relatively well, though informally in some respects. 

There !s no organization chart in existence for I JA. There are several documents 
available which spell out the missions, goals and objectives of IJA and, to some extent, 
the Appellate Judges Seminars. 

Fiscal procedures are governed by NYU (including accounting, purchasing, and 
auditing procedures). Sound policies exist in most financial areas although the cur- 
rent statu;, reports of LEAA grant spcndir.g were not available at the time of our visit. 

IJA. adheres to NYU's personnel policies and procedures, at least for the nonpro- 
fessional staff. Position descrir' ion-^ on file were three years old, and th»re are only in- 
fonnri3l procedures utilized fo grievances and disciplinary actions. However, these are 
relatively minor deficiencies. The overall impression is that IJA's small staff operates 
efficiently and that the noninstructional aspects of the seminars are also well planned 
and managec 



Challenges and 

Future Directions 



While funding is mention^H as a concern, past and present directors of IJA dif 
fer as to the seriousness of the problems caused by recent LEAA funding reductions 

H t nnL; ^ T-^ I t'^^ll ^ P''^'^'^"^ f^'' ^he future, assuming no ad- 

ditional reductions of size by LEAA. 
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Training Objectives/ 

Perceived Effectiveness 

The objectives of the IJA Appellate Judges Seminars and the instmctors who 
present them are in concert, and the instructors perceive that they are very successful 
in achieving their higher priorities, (see Table 8A.) 

Seminar participants agree that the seminars are relevant to them and their courts, 
giving the seminars the highest rating possible. 

Table 9A offers further evidence of the effectiveness of the seminars, as the parti- 
cipants and the instructors judge them. Only in the a^ea of making organizational 
changes as a rt . jIt of training, where 50 percent have p;ctempted, is the effectiveness of 
the seminars less impressive though still noteworthy. 

As for the questions concerning a return to the seminars in the future, it should 
be noted that these programs are intended primarily for new int^rmcxliate and senior 
':ppellate judges. It would be unlikely for a judge to return to the seminars unless 
le/she had attended an intermediate seminar and had moved to the senior court or 
court of last resort. 

Table lOA displays the responses of both participants and instructors to questions 
concerning the soundness of the training design. The least favorable responses come 
from both participants and instructors on questions 6 and 7, concerning the degree to 
which training provides opportunities to practice what is taught, and the degree to 
which participants are kept informed of their progress during the seminars. 



Training Setting, 

Techniques and Faculty 

The Appellate Judges Seminars receive very high ratings from participants and 
instructors in matters concerning the training setting {Table llA), and from instructors 
regarding the management of the seminars {Table 12A). 

Participants and instructors agree ^^at lectures and group discussions constitute 
the primary teaching methods utilized {Table ISA), and the training participants say 
they find these tv o techniques extremely useful, along witi. the informal conversations 
that take place outside the classroom setting. 

The participants give extremety high ratings to the instructors in the categories of 
substantive knowledge, practical experience and teaching ability, but they almost never 
call upon instructors after the seminars conclude for follow-up or technical advice 
{Table 14A), Interestingly, instructors say they are called upon more often by former 
participants. The seven instructors who were surveyed have particip ,ted in the seminars 
a subrrtantial number of times {Table 15A). 
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TABLE 8A 

INSTRUCTORS' PERCEPTIONS OF LEARNING OBJECTIVES (UA) 



Objective 



6. 



Enhance participants' under- 
standing of their roles 

Improve participants' pro- 
ficiency of their roles. 

Update and increase partici- 
pants' substantive k^iowledge. 

Enable participants to intro- 
duce new techniques and pro- 
cedures in their systems. 

Increase communication and 
consultation among profes- 
sional pe>ers/colleagues. 

Enable participants to in- 
fluence/promote change 
among other courts person- 
nel. 



Perceived Achievement 

Average Category of 

Priority Rating Median 
n-High;7>Low) Response 



1,83 



Very well 



2.17 
2,33 



Very well 
Fairly well 



3.83 



Fairly well/ 
Very 



4,17 



Fairly well 
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TABLE 9A 
MULTIPLIER EFFECTS (IJA) 



Aspect 



Type of 
Respondent 



Number/% 
Responding 



Frequency of Response 



Wish to return to 
institute. 

Perception of par- 
ticipants' desire to 
return to institute. 

Perception of par- 
ticipants' actual 
return to institute. 

Share institute 
materials. 

Perception of par- 
ticipants' use of 
institute materials. 

Use institute materials. 

Attempt to make insti- 
tute-spurred per- 
sonal changes. 

Attempt to mske insti- 
tute-spurred organi- 
zational changes. 

Recommend institute 
to others. 



Participant 

Instructor 

Instructor 
Participant 

Instructor 
Instructor 

Participant 

Participant 
Participant 



H JL YES NO No response 



14 100 10 71 4 29 



6 86 4 67 2 33 



3 43 - 



3 100 



14 100 8 57 6 43 

6 86 6 100 - - 

7 100 7 100 - 

14 100 11 79 3 21 

14 100 7 50 7 50 

14 100 14 100 - 
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TABLE lOA 
SOUNDNESSOF TRAINING DESIGN lIJA) 



Category _ 

Type of Number of Median — Frequency of Response 

Training Desifln Component Respondent Responding Response^ Not at all Somewhat Fairly well Very well Thorougiily No Response 

1. Degree based on Participant 13 Very well 
profession's needs. (93%) 

Instructor 5 Very well 
(71%) 

2. Degree individual's Participant 14 Fairly well 2 1 
needs are considered. (100%) (14%) jyo^j 

Instructor 5 Vei^well 
(71%) 

3. Degree expected out- Participant 14 Fairly well 2 1 
r comes presented at (100%) (i4%| |7%| 
I outset of course. Inst, ctor 4 Fairly well 1 
0 (57%) (25%) 

4. Degree learning ob- Participant 13 Very well 1 1 
jectives clear and (93%) (7%) (7%) 
succinct. Instructor 5 Fairly well 

(71%) 

5. Degree objefitives are Participant 14 Somewhat 4 5 
demonstrabli!. (100%) (29%) (36%) 

Instructor 5 Very well 
(71%) 

6. Degree training Participant 13 Somewhat 5 2 
provides opportunities (93%) (38%) (15%) (15%) (15%) (15%) ' V: 
to practice what is Instructor 5 Somewhat 1 4 
taught. (71%) (2o%| 



1 


6 


6 


1 


(7%) 


(46%) 


(46%) 




1 


3 


1 


2 


(20%) 


(60%) 


(20%) 




5 


5 


1 




(36%) 


(36%) 


17%) 




1 


3 


1 


2 


(20%) 


(60%) 


120%) 




5 


2 


4 




(36%) 


(14%) 


(29%) 




L 


1 




3 


(50%) 


(25%) 






3 


6 


2 


1 


(23%) 


(46%) 


(15%) 




3 


2 




2 


(60%) 


(40%) 








3 


2 






(21%) 


(14%) 




1 


4 




2 


(20%) 


(80%) 






2 


2 


2 


1 


(15%) 


(15%) 


(15%) 










2 
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TABLE 10A (Continued) 



Category 

Type of Number of Median Frequency of Response 

Training Design Component Respondent Responding Response Not at all Somewhat Fairly well Very well Tfioroughly No Response 



7. Degree participant 


Participant 


12 


Not at all 


11 








1 


2 


informed of progress. 




(86%) 




(92%) 








(3%) 






Instructor 


2 


Not at all/ . 


1 


1 








5 






(29%) 


Somewhat 


(50%) 


(50%) 










8. Degree objectives, 


Participant 


14 


Very well 






2 


6 


6 




materials, and activities 




(100%) 








(14%) 


(43%) 


(43%) 




cohesive. 


Instructor 


5 


Very well 


1 




1 


3 


- 


2 














(20%) 


(60%) 






9. Degree feedbacl(/ 


Participant 


13 


Fairly well 


4 


1 


2 


1 


5 


1 


evaluation useful. 

mm 




(93%) 




(31%) 


(7%) 


(15%) 


(7%) 


(39%) 




" 


Instructor 


4 


Somewhat/ 


1 


1 


1 


1 




3 


mi 
mi 

a 




(57%) 


Fairly well 


(25%) 


(25%) 


(25%) 


(2b%) 






10. Degree of opportunity 


Particvant 


13 


Thoroughly 


1 


2 




2 


8 


1 


to provide input to 




(93%) 




(7%) 


(15%) 




(15%) 


(61%) 




faculty. 




















11. Degree training up- 


Instructor 


5 


Very well 






1 


3 


1 


2 


graded and modified 




(71%) 








(20%) 


(60%) 


(20%) 





in relation tocurrtnt 
state of the art. 



TABLE 11A 

ADEQUACY OF THE TRAINING SETTING (IJA) 



Type Of Number/% 
Aspect Respondent Responding 

1. Similarity of participants' Participant 14 

; jles/experience ( 1 00%) 

Instructor 7 

(100%) 

2. Suffciently small classes Participant 14 

(100% 

Instructor 7 

(100%) 

3. Sufficiently long courses Participant 14 

(100%) 
Instructor 7 

(100%) 

4. Match betweei, partici- Participant 14 
pant needs and instruc- ( i oo%) 
tor expertise Instructor 7 

(100%) 

5. Availability of instruc- Participant 14 
tors outside of class ( 1 oo%) 

Instructor 7 

(100%) 

6. Sufficient number of Participant 14 
contact hours between ( 1 00%) 
participants and in- Instructor 7 
structors (100%) 

7. Reinforcement of be- instructor 5 
havior/skills (71%) 

8. Adequati^ support ser- Participant 14 
v'ces (100%) 

Instructor 6 
(86%) 

9. Adequate physical ac- Participant 13 
commodations (93%) 

instructor 7 

(100%) 



Frequency of Response 



No 



1 o?> 


Mn 


Kesponse 


13 


1 


— 


(93%) 


(7%) 




7 


— 




(100%) 






13 


1 


— 


(93%) 


(7%) 




7 




— 


(100%) 






13 


1 




(93%) 


(7%) 




7 




— 


(100%) 






13 


1 


— 


(93%) 


(7%) 




7 




— 


(100%) 






13 


1 


— 


(93%) 


(7%) 




7 






(100%) 






13 


1 


— 


(93%) 


(7%) 




7 






(100%) 






4 


1 


2 


(80%) 


(20%) 




14 






(100%) 






5 


1 


1 


(83%) 


(17%) 




13 




1 


(100%) 






7 







(100%) 
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TABLE 12A 
MANAGEMENT OF TRAINING EVENTS (IJA) 



Aspect 



1 . Clarity of lines of authority/ 
responsibility 

2. Adequacy of administrative 
procedures 

3. Instructors appraised of per- 
formance/expectations 

4. Instructon given orienta- 
tion/staff training 

5. Progra > modification based 
on assessment 



CTP Instructor Respon se 

Nun^ber 

Responding Yes No 



!L Ji ii % N % 

7 100% 7 100% 

6 86% 6 100% - 

7 100% 7 100% 

7 100% 6 86% 1 14% 

7 100% 6 86% 1 14% 
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TABLE 13A 
FREQUENCY OF METHODS USED (UA) 



Method 



IV?3dian Response 



CTP Participant 



CTP Instructor 



1. 


Lectures 


Most of the time 


w 1 ten 


2. 


Discussion groups 


Lften 


Often 


3. 


Panel discussions 


Not at all 


Often 


4. 


Case studies 


N/A 


Often 


5. 


Role playing, simulation 


Not at all 


Not at all 


6. 


Videotaping 


N/A 


Not at all 


7. 


Moving pictures 


N/A 


Not at all 


8. 


On-the-job training 


N/A 


Not at all 


9. 


Computer Assisted 
Instruction 


N/A 


Not at all 
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TABLE 14A 
PERCEIVED FACULTY COMPETENCE (IJA) 



Category Totally Minimally 
Number of Median Unknow- Knowledge- Quite No 

Aspect Responding Response ledgeable able Adequate Knowledgeable Expert Response 



Degree of their sub- 14 Expert _ - _ 4 10 

stantive knowledge. (100%) (29%) (71%) 



Extensivenessof 

their practical ex- 14 Expert - - - 2 12 

perience. (100%) (14%) 



Teaching ability. 12 Quite - - 2 7 3 2 

(86%) Knowledgeable (17%) (58%) (25%) 

Not at Once or No 
all twice Periodically Often Continually Response 

Number times 

called upon staff/ 13 Not at all 11 2 - - - 1 

instructors since (93%) (85%) (15%) 

training for followup 
technical assistance. 



TABLE ISA 

FACULTY FOLLOWUP/CONT. RELATIONSHIP (IJA) 

Called upon by CTP 

NuTin^H^r9f TiTTi^ Participants Invited back to In stitute 

N % N i; 

None 2 33 0 0 

I- 5 ■ 1 17 3 43 
6-10 2 33 3 43 

II- 25 1 17 1 14 
Greater than 25 0 0 0 0 



INTEREST IN TEACHING AGAIN 
YES (%) NO (%) 

100 0 
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2. APPELLATE JUDGES' CONFERENCE 
AMERICAN BAR ASSOCIATION 



The subject of this section is the Appellate Judges' Conference (AJC) of the 
American Bar Association (ABA) and, specifically, the Project for Continuing Appel- 
late Education, The purpose for documenting these characteristics is to note features 
of the training process that potentially influence outcomes, to consider the project's 
responsiveness to its training goals and the perceived needs of its constituency, and to 
consider the nature and quality of the management and administration which support 
the program. 

The description of the project, and its sponsor, the Appellate Judges' Conference, 
Is based primarily on information obtained on a visit to the conference in Chicago on 
December 13 and 14, 1978. The data collection was conducted by a three-person team 
and included interviews with the Director, Division of Judicial Service Activities; the 
Program Director; the Staff Director for Grant Fund Administration, Fund for Public 
Education of the ABA; the Assistant Program Director; and the Administrative Assis- 
tant 

During the visit, the research team was introduced to the programs and facilities 
of the conference. The team was also oriented to the resources available to the confer- 
ence as a result of its affiliation with the ABA, as well as its responsibilities resulting 
from this relationship. All documents and records from the conference were made 
available to the research team, including records of participant evaluations, curriculum 
materials, and policy and procedures manuals. 



History of the 
Project 

The Project for Continuing Appellate Education is beginnir^g its eleventh year of 
providing continuing legal education for state and Federal appellate judges. It origina- 
ted from the strong desire of a group of appellate judges to improve the quality of their 
system. 

in 1968 a group of appellate judges gathered together and discussed how to go about tm* 
proving the quality of the nation's appellate system. A number of suggestions were enter- 
tained. The one that received a consensus concerned continuing the judge's education 
after elevation to the appellate bench, (t was concluded that the need to keep up with the 
rapidly changing law that each was required to administer compelled a decision to under- 
take a seminar program on a national basis offering courses designed for the appellate 
judge. With a small grant from the Law Enforcement Assistance Administration, a modest 
program was begun. (Final Grant Report, Appellate Judges' Conference, December 
1978.) 



The Appellate Judges' Conference of the American Bar Association has sponsored 
this project since 1971; it has been supported by discretionary funds since 1973 under 
the Judicial Training Package grants. Until 1976, the project was coordinated on behalf 
of the conference by the Institute of Continuing Legal Education at Locisiana State 
University. From 1976 to the present, the project has been sponsored directly by the 
Appellate Judges' Conference of the American Bar Association in Chicago. 

The Appellate Judges' Conference is a membership organization of the American 
Bar Association. As such, it is subject to the policies and procedures of the ABA, its 
board of governors, and house of delegates. Organizationally, the AJC is overseen by 
the Judicial Administration Division of the ABA. 

ADMINISTRATION AND MANAGEMENT 

The conference is further governed by its bylaws, which clearly outline the admin- 
istrative body, its composition, role, and responsibilities. Additionally, the bylaws spec- 
ify the qualifications for membership in the conference, members of courts of last re- 
sort or intermediate appellate courts and former appellate judges. Approximately 700, 
or 70 percent, of the appellate judges in the United States are currently members of 
the conference. 

There are four officers of the conference: chairman, chairman-elect, vice chair- 
man, and secretary. These officers are annually elected by the membership. 

The executive committee consists of the officers, a delegate to the ABA's house 
of delegates, the immediate past chairman, and seven members of the conference who 
are elected by the other members. The executive committee is the policy-setting body 
of the conference and directly oversees the affairs of the conference and its projects. 

The day-to-day management and program development of the conference are the 
responsibility of the program director and his staff. While overall management is 
governed primarily by the policies and procedures of the ABA and those set by the 
executive committee^ the development of the Project for Continuing Appellate Educa- 
tion (scheduling, curriculum, topics; materials development, etc) is manaoed directly 
by the program director. Additionally, the Project on Continuing Appellate Educa- 
tion utilizes the guidance of the conference's Committee on Continuing Appellate 
Education and an academic consultant to guide format, development of seminar topics, 
selection of faculty and lecturers, and content of the educational materiaSs. 

SOURCES OF FUNDING 

The Project for Continuing Appellate Education currently receives funding from 
four sources, LEAA and three private foundations. 
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In the 1979 program year, for the first time, project income will also be generated by 
tuition charged to the participants. 

RELATIONSHIP WiTH RELATED PROGRAMS 

The Appellate Judges' Conference and the Institute of Judicial Administration, 
based at New York University, work together in providing education to the appellate 
judiciary. The reli7itionship, as stated by the conference, is complementary rather than 
one of competition. The Institute of Judicial Administration program is directed at 
new appellate judges, while the confr^rence program provides in>depth and updated in- 
formation on a continuing basis. This relationship is less formal today than it once was, 
when both programs were part of the six-institution Judicial Training Package; how- 
ever, it is seen as no less important in guiding the direction of the AJC's Project on 
Continuing Appellate Education. 

The conference also provides assistance to in-state training efforts on an as-needed 
basis. !t should be noted in this regard that few states provide appellate education due 
to the small number of appellate judges in each state. 



Gocds and 

Objectives 

The Project on Continuing Appellate Education has sought to improve the qual- 
ity of the nation's appellate justice system by providing employees of state and Federal 
appellate courts with information that will enable them to improve the quality of ser- 
vice offered to the public. Specific objectives are: 

• To expose the maximum number of appellate judges to the educational 
activities of the profect. The goal is to have between 20 and 30 percent 
of the active members of the nation's appellate courts attend during the 
academic year. 

m To conduct an annual seminar that exposes appellate court clerks to 
new and improved methods of administering the appellate courts, 

m To expose appellate central staff attorneys to an educational program 
similar to that for the judge, but designed specifically for the central 
staff attorney. 

m To develop text materials to support the seminar that would be used 
both at and after the seminar 

m To plan and implement special educational programs for the appellate 
system. 
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• To obtain a candid evaluation of the educational program to see wheth- 
er it is effectively and efficiently pursued. 

These objectives are implemented by the project with a central theme in mind: to 
expose as many of the appellate judiciary as possible to substantive and procedural as- 
pects of the court, mcluding introducing the judiciary to technical issues and philoso- 
phies which are essential to the decision-making process. The project staff sees its role 
as one of becoming a center of continuing appellate education and filling the continu- 
ing needs of the appellate courts, thereby helping to solve the administrative problems 
of the courts. 



Programs 



TRENDS 



The Project on Continuing Appellate Education has remained essentially un- 
changed in objective and focus since its inception. Over the years, some changes have 
occurred in the number and types of programs offered. In order to accommodate the 
growing number of applicants, the number of appellate judges' seminars has increased 
from five to six per year. Additionally, the training provided to appellate law clerks 
has changed. This program, begun in 1977 as part of the Appellate Judges' Seminars, 
expanded considerably in 1978 with the creation of a model program for appellate 
law clerks. 

Another change is the development of a seminar for appellate court clerks in 
1976. Also, a program for central staff counsel was conceived and is currently conduc- 
ted in conjunction with the annual meeting of the American Bar Association. The fol- 
lowing table displays the project's focus and changes for the past five years. 



Program Charactcrirtfo* 



PROJECT TRENDS 
1974 1975 1976 



1977 



1978 



Goals 

Number of Programs Offered 

Length of Program 

Average Number of Partici- 
pants (each program) 

Target Audience 



To educate as many appellate judges, appellate law clerks, appellate 
central staff counsel, and appellate clerks as possible at the finest edu- 
cational program that is practical. 



6 

4 days 

23 

Appellate 
Judges, 
Law Clerks 



* Responses reflect predominent categories or Issues. 



4 

4 days 

26 

Appellate 
Judges, 



8 

4 days 

33 

Appellate 
Judges, 



8 

4 days 

36 

Appellate 
Judges, 



8 

4 days 
37 

Appellate 
Judges, 



Uw Clerks Law Clerks Law Clerks Law Clerks 
Court Clerks Court Clerks Court Clerks 
Central Staff Central Staff Central Staff 
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CURRENT ACTIVITIES 



Currently, the Project on Continuing Appellate Education conducts four educa- 
tional programs: appellate judges' seminar series, appellate law clerk program, seminar 
for appellate court clerks, and a meeting of appellate central staff attorneys. The table 
entitled, "Programs of the Project on Continuing Appellate Education," lists activities 
occurring from January 1977 to September 1978, 

Appellate Judges' Seminar Series 

This program forms the cornerstone of the Project on Continuing Appellate Edu- 
cation. The program is designed to keep appellate judges current on substantive and 
procedural mattsrs relative to appellate courts. 

The curriculum is planned on the premise that new developments in appellate jus- 
tice occur constantly and that the seminars will meet judges' needs for new knowledge 
on a continuing basis. The seminars offer an equal amount of exposure to new tech- 
niques, procedures, and issues of law. These include impact decisions by the U.S. Su- 
preme Court, recent developments in the Law of Products Liability, recent develop- 
ments in Collateral Attack, and the use of techniques to avoid repetitive litigation. 
Topics of substantive law vary from seminar to seminar and year to year, depending on 
the most pressing issues at the time. Topics of particular importance are often repeated 
at several seminars in a single academic year in order to make the topic available to as 
many appellate judges and justices as possible. 

Seminar topics which address new techniques and procedures also vary. Again, as 
recent developments are known and tested, they are incorporated into the curricula. 

Text materials are developed to support the presentation and to be used follow- 
ing the seminar These consist of articles and papers by leaders in the field, written ap- 
pellate opinions, names and addresses of seminar participants, and other reference ma- 
terial pertinent to the seminar topics. 

Appellate Law Clerk Program 

The Appellate Law Clerk Program is designed to train newly appointed law clerks 
to better assist appellate judges. A model program has been created to orient the law 
clerks to general issues and procedures. At the same time, the model is designed so that 
it is readily adaptable to meet the needs of individual jurisdictions. In this way, the 
program can be provided to law clerks in their home jurisdictions for less than the cost 
of bringing the clerks to a national program, according to the project. 

As part of the curriculum, the AJC has developed a manual which may be modi- 
fied to reflect local rules, laws, and customs 

Appellate Court Clerks Seminar 

The program for appellate court clerks consists or a seminar conducted during the 
annual meeting of the National Conference of AppH;!late Court Clerks (NCACC). The 
planning of the seminar is done by the NCACC with assistance from the Committee on 
Continuing Appellate Education. 
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PROGRAMS OF THE PROJECT ON CONTINUING APPELLATE EDUCATION 



ACTIVITY 


DATES 


FORMAT 


TOPICS 


PARTICI' 
PANTS 


TARGET GROUP 


1977 Appellate Judges' 
Seminar Series 


Miami 
January 17-20 


Seminar 


Impact Decisions/Recent Developments 
/Judicial Review of Decisions 


33 


Statfi fViiirT Annellate 

Judges 




New Orleans 
March 27-31 


Seminar 


Impact Decisions/Recent Develop- 
ments/Judicial Relations with the Bar/ 
LEAA and the Courts 


23 


State Court Appellate 
JudciRs 




Tucson 
April 17-21 


Seminar 


Impact Decisions/Recent Develop- 
ments/Standards for Appellate Courts/ 
Judicial Philosophy 


18 


State Court Appellate 




Williamsburg 
May 22-26 


Seminar 


Appellate Administration Techniques/ 
Recent Developments/Impact 
Decisions 


32 


State Court Appellate 
Judges 




La? Vegas 
September 25-29 


Seminar 


Recent Developments/Appellate Inno^ 
vations/Judicial Discipline/Impact 
Decisions 


23 


State Court Appellate 
Judges 




San Francisco 
October 23*27 


Seminar 


Impact Decisions/Recent Develop- 
ments/Methods for Appellate Efficiency 


60 


State Court Appellate 
Judges 


1978 Appellate Judges' 
Seminar Series 


Tucson 
March 19-23 


Seminar 


Impact Decisions/Medico- Legal Prob- 
lems/Techniques in Common Law 
Cases 


16 


State Court Appellate 
Juriaes 

I 




San Diego 
April 16-20 


Seminar 


Impact Decisions/Economics and the 
Law/Appellate Efficiency /Powers of 
Courts 


25 


State Court Appellate ! 
Judges | 

i 




Williamsburg 
May 28-June 1 


Seminar 


Pre-Argument Settlement Conference/ 
Aoolication of New Rules/Currfint 
Developments 


29 


t 

State Court Appellate j 

! 


t 


Boston 
September 24*28 


Seminar 


Opinion Writing/Judicial Philosophy/ 
Impact Decisions/Recent Develop- 
ments 


42 


i 

State Court Appellate i 
Judges j 

i 




San Francisco 
December 22-26 


Seminar 


Impact Decisions/Recent Develop- 
ments/Docket Procedures 


57 


State Court Appellate j 
Judges i 

! 

I 
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PROGRAMS OF THE PROJECT ON CONTINUING APPELLATE EDUCATION - Contlnuad 



ACTIVITY 


DATES 


FORMAT 


TOPICS 


PARTICI- 
PANTS 


TARGET GROUP 


1978 Appellate Law 
Clerks Program 


Summer/Fall 
Florida 


Seminar 


Law Clerks/Procedures/Litigation 
Process 


Mnt PrO- 

vided 


Appellate Law 
Clerks 




Wisconsin 


Sleminar 


Functions of Aooellate nniirts/Pnlp nf 
Law Clerks/Procedures/Litigation 
Process 


vided 


Appellate Law 
Clerks 




South Dakota 


Seminar 


Functions of AoD^llate Court^/Rolp 
of Law Clerks/Procedures/Litigation 
Process 


Mnt Pro- 

vided 


Oiaic anu ^uerai 
Appellate Law 
Clerks 




Arizona 


Seminar 


Functions of ADDpll;)tP fVitirtc/Rnlp 

of Law Clerks/Procedures/Litigation 
Process 


Nnt Prn- 

vided 


ovale aiiu 1 euci Jli 

Appellate Law 
Clerks 




Rhode Island 


Seminar 


Functions of Appellate Courts/Role 
of Law Clerks/Procedures/Litigation 
Process 


Not Pro- 
vided 


State and Federal 
Appellate Law 
Clerks 




New Hampshire 


Seminar 


Functions of Appellate Courts/Role 
of Law Clerks/Procedures/Litigation 
Process 


Not Pro- 
vided 


State and Federal 
Appellate Law 
uierKS 


1977 Appellate Court 
Clerks Seminar 


Coeur d'Alene 
July 28-August 1 


Seminar 


Educating the Public/Automation of 
Records/Standards for Courts/ 
Security/Personnel Trends 


64 


State and Federal 
Appellate Court 
Clerks 




Mackinac Island 
August 13-17 


Seminar 


Records Management/Expediting 
Workload/lnnovations/Review of 
Williamsburg Seminar 


68 


State and Federal 
Appellate Court 
Clerks 


1977 Appellate Central 
Staff Attorney Meeting 


Chicago 
July 


Meeting 


Appellate Procedures/Role of Staff 
Counsel/Effective Writing 


42 


State and Federal 
Appellate Central 
Staff 




New York 
June 


Meeting 


Analyzing Appellate Procedures/Role 
of Staff Counsel/Effective Writing 


22 


State and Federal 
Appellate Central 
Staff 
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The program is tailored to the ongoing challenges and problems facing appellate 
court clerks. Essentially, it offers expa^ure to and instruction in new methods of ad- 
ministering the appellate courts, introduction to new equipment, and the opportunity 
to interact with peers in the profession. 

Appellate Central Staff Attorneys 

The Appellate Judges' Seminar Series has, in the past, espoused the concept that a 
central legal staff may assist the appellate courts. Consequently, use of this new admin- 
istrative role has grown. In response to this phenomenon, the Project on Continuing 
Appellate Education added a program in 1976 designed specifically for appellate cen- 
tral staff counsel. Currently, a meeting is held, in conjunction with the annual meeting 
of the American Bar Association, to identify problems in their systems and to shape 
their role to meet the needs of their systems. 

As with the project's other programs, the topics for the meeting are updated and 
charged to meet continuing needs. 

NEEDS ANALYSIS 

The project utilizes a relatively informal, continuing process for assessing the 
rneeds of its target audience. Members of the educational committee are a major source 
of information regarding needs by virtue of their positions in the nation's appellate 
courts. The program director says he remains in contact with members of appellate 
courts, maintains liaison with many state judicial education programs, conducts a review 
of relevant literature such as the National Manpower Survey of the Criminal Just ice Sys- 
tem, and utilizes other neces^ry means of determining needs. The nation's appellate 
judges provide information regarding their educational interests and needs at the semi- 
nars. Participant evaluations and informal discussions provide insights into the needs of 
the appellate courts. 

CURRICULUM DEVELOPMENT 

The curriculum development process is tied to the mechanisms for assessing 
needs. The Committee on Continuing Appellate Education meets with 5ts academic 
consultant and the program director and develops a list of topics for the upcoming 
seminars. The group attempts to strike a balance between areas of substantive law 
that appear to be undergoing change, administrative techniques that might equip the 
courts to handle expanding workloads, and topics covered in prior seminars that 
received special interest. 

Based on the topic list created at the meeting, the program director, the academic 
consultant, and the chairman of the committee develop the actual curriculum and fac- 
ulty for each seminar. During this process informal discussion takes place with past and 
prospective seminar participants to assess interest in the topics. In this way, the cur- 
riculum is more likely to focus on the immediate needs of the appellate courts. Fur- 
thermore, the project has an abiding concern to make the participants feel that it is 
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"their" program. The project feels that, together with the various means of staying in 
touch with the participants' needs, the curriculum development process assures this 
identity with the program. 

Two months prior to each seminar, each faculty member submits an outline of 
the topic to be presented. Once the outline has been approved by the program director 
and educational committee, the project staff compiles materials provided by the fac- 
ulty, as well as other pertinent documents, into the seminar text. 

A major variation of this process is found in the development of the appellate 
court clerks seminars. The National Conference of Appellate Court Clerks plans the 
program with assistance, as requested, by the Committee on Continuing Appellate Edu- 
cation. However, the committee reserves the right of final approval of the program. 

One other exception to this process occurs when the Appellate Judges' Confer- 
ence is invited to participate in educational programs of states or other national train- 
ing programs. In these instances, the requesting agency determines the needs and iden- 
tifies the desired objectives and curriculum. 

The basic format ijsed by the project is also determined by the Committee. The In- 
structional method is a four-day, lecture/discussion based on the following rationale: 

» The seminars are ''updatmg" an already vast body of knowledge, not 
servicing inexperienced individuals, 

■ The seminars are designed to present ideas and generate peer exchange, 
thereby making the information directly relevant 

■ The participants have little time tc spend away from their courts; there- 
fore, the program must be concise and to the point if it is to attract par- 
ticipants. 

FACULTY SELECTION 

The principal, criterion employed in selecting faculty is in-depth knowledge of a 
topic area. The project attempts to find the best informed individuals. Other critera 
used in the selection include: 

■ The ability to relate knov^ledge to the constituency in an interesting 
way; 

• Willingness to work with the project in designing the program; and 
» Availability. 
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Faculty members are recruited, as needed, by the educational committee, the pro- 
gram director, and the academic consultant. This group utilizes all available sources to 
identify potential faculty: former participants, other faculty, and national organiza- 
tions and institutions. 

A profile of faculty members derived from institute records Is: 



Average Age 

Current Position 

Years of Legal Experience 

Years as Legal Educator 

Typical Degree 

Areas of Expertise 



52 years 
Appellate Judge 
26 years 
12 years 
LLM, LLD, JD 

Constitutional Law, Criminal Law and 
Procedure, Law and Medicine, Con- 
tracts, Products Liability, Judical Ad- 
ministration, Insurance, Civil Law, Jn- 
international Legal Methods, Appel- 
late Law and Procedures, Evidence 



Average Tenure with Project : 3 years 



PERMANENT STAFF 



The permanent staff positions devoted to the programs of the Project on Continu- 
ing Appellate Education, as of the time of our visitation, are listed below. 

■ The Program Director is responsible for overall program development 
and administration of the project. 

• The Assistant Program Director assists in all phases of program develop- 
ment and daily administration of the project 

• The Administrative Assistant is responsible for all administrative sup- 
port functions of the project including registration, recordkeeping, ar- 
rangements for seminar facilities, and the like. 
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In addition to the full-time project staff, support is provided by an academic con- 
sultant and a law clerk, described below. 



■ The Academic Consultant is an esteemed professor of law with approx- 
imately 46 years'' experience as an educator. His expertise reaches into 
most specialties in legal practice, such as civil law and procedures, 
juvenile justice, and appellate law, 

■ The Law Clerk is utilized primarily to perform legal research in support 
of text materials for the programs. 

The following tablr» prc-sent* the proportion of time devoted to the various admin- 
istrative and program d' , -ii^^ment tasks of the project. 



STAFF TIME ALLOCATION 
(Percent) 



Activity Area 



Staff Member 


Curriculum 
Administrative Development 


Materials Conf. Mgmt./ 
Marketing Development Participation 


Other 


Total 


Program Director 


6% 


14% 


74% 


4%% 


10% 


65% 


100% 


Assistant Program Director 


35% 


25% 


0 


25% 


12% 


3% 


100% 


Administrative Assistant 


60% 


0 


0 


30% 


0 


10% 


100% 


Law Clerk 


0 


0 


0 


100% 


0 


0 


100% 


Academic Consultant 


0 


100% 


0 


0 


0 


0 


100% 



Effectiveness 

This section deals primarily with the project's perspective of its success and assess- 
ment of its goals and objectives. Additionally, previous outside evaluations of the proj- 
ect are discussed. 

MARKETING OF PROGRAMS 

Informal marketing of the programs occurs through word of mouth of past par- 
ticipants. The philosophy underlying this approach is that a quality program will sell 
Itself, The project feels its reputation is positive; therefore, participants promote the 
program and encourage participation by colleagues. 
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Formal marketing conducted by the projiK:t is through an annual seminar an* 
nouncement. At the beginning of each academic year a brochure announcing the sched- 
ule for the appellate judges' seminar series is sent to every state and Federal appellate 
judge. This brochure contains a statement regarding LEAA and other sponsorship* 

Pubiic relations for the project, such as press inquiries, is solely the responsibili- 
ty of the communications department of the ABA. 

SELECTION OF PARTICIPANTS 

Participants are selected on a first come, first served basis. However, when it is 
necessary to limit the number of applicants, such as with the judges' seminars, new ap- 
plicants receive priority. The judges' seminars attempt to limit participation to 40 jud- 
ges at each seminar. 

ASSESSMENT OF THE PROGRAMS 

The formal assessment process utilized by the project involves the use of an exten- 
sive program evaluation form distributed to all seminar participants (the process applies 
only to the appellate judges' seminars). 

The evaluation focuses primarily on the process of training; the environment, de- 
livery, faculty, format, topics covered, and the like. Responses are strictly voluntary. 
There has been a 51 percent response rate for the 1977-75 seminars. 

These evaluation forms are used for two purposes: to record recommendations 
and to solicit the participant's input and suggestions for improving the program. Fol- 
lowing each seminar, the responses are analyzed and a summary is developed for use by 
the program director and educational committee in developing future programs. 

The following is a summary of the comments from the in-house program evalua- 
tion for the 1977-78 grant period. 

General Comments 



Accommodations were considered good to excellent. 

The value of hospitality sessions wsj rated good to excellent. 

The quality of presentations, coverage of topics, and quality of mate- 
rials were considered good. 

Suggestions regarding accommodations were concerned with proximity 
to restaurants and recreation, availability of tourist information for 
spouses, and better facilities at the hotel. 
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■ Suggestions for improving the curriculum were concerned with time al- 
lotments for various topics, the balance between substantive and proce- 
dural topics, and the addition of new topiics. 

■ Approximately 95 percent of the respondents indicated the informal 
method of presentation was preferred, 

■ The preferred length of time devoted to each topic was three hours. 

■ Various topics and speakers were suggested for future seminars; opinion 
writing was among the most frequently requited topics. 

■ The four-day seminar span was preferred. 

■ Spring and autumn months were the most desirable seminar periods. 

■ Various locations were suggested as future conference sites. Typically, 
they were large metropolitan areas, easily accessible by public transpor- 
tation, wffere leisure time and sightseeing activities are available for 
spouses. 

■ OnJy modest tuition funds will be made available to participants by 
their courts. 

In addition to the formal evaluation process, the program director and at least 
two members of the educational committee observe each seminar. Faculty members 
are also consulted informally regarding program improvements. 

The program director Indicated that a proposal was made to the educational com- 
mittee regarding the use of an outside evaluator in assessing the program. The commit- 
tee rejected the proposal as unnecessary. 

PROGRAM BENEFITS 

Although no empirical data existed and no formal means for long-term follow-up 
is utilized, the project has cited several benefits believed to be the direct result of the 
educational program. This information is received by the project through word of 
mouth and letters from participants. Additionally, the seminar management records 
provide some of this information. Some benefits cited by AJC staff are: 

« Over 30 percent of the nation's appellate judges, 60 percent of the na- 
tion's appellate court clerks, and a large proportion of its appellate cen- 
tral staff counsel attended at least one seminar in the 1977-78 grant pe- 
riod. 
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■ The project feels there is better understand/ng of charjging fields of law. 

« New ideas regarding administration of appellate courts have been shared 
and originate from the discussion. 

m The interpretation of the law has tended to become more uniform. 

m New concepts of law have developed. 

m Lateral communication exists, for the first time, among appellate jud- 
ges. 

m The use of appellate central staff has grown as a result of the seminars. 

OUTSIDE. EVALUATIONS 

In 1974 the National Center for State Courts evaluated the Appellate Judges' 
Seminar Series, then based at Louisiana State University, as part of its assessment of 
the Judicial Training Package grants* A key finding cited In the evaluation was that the 
program appeared to be the only group serving appellate judges on a continuing basis. 
Furthermore, coordination with state programs was not a problem, due to a lack of 
programs of similar focus. 

The recommendations centered on the process used in developing the seminars: 
depth of the curriculum advisory committee and broadening of the evaluation process. 
The need for greater representation of minority groups on the faculty and considera- 
tion of experts from nonlegal disciplines as faculty members were recommended. An- 
other significant recommendation was that the National Center sponsor long-term stud- 
ies leading to the creation of a single national training center for appellate judges. 

A word of caution is necessary in reviewing this section. This evaluation refers to 
the program as it existed at LSU. These findings and recommendations are not neces- 
sarily applicable to the current program. 



Other Institutional 
Activities 

In addition to the seminars, the Appellate Judges' Conference is engaged in a 
variety of other projects, many of which complement the work of the Project on Con- 
tinuing Appellate Education. 
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RESEARCH AND PUBLICATIONS 



Legal research Is an integral part of the project's activities. The law clerk and pro- 
gram director monitor U.S. Supreme Court decisions, new procedures implemented by 
state appellate courts and the like, and prepare material for inclusion in the seminar 
texts and law clerk's manual. 

EXTERNAL RELATIONS 

The project staff receives numerous invitations from state-administered appellate 
education programs for new'y appointed judges, appellate court clerks, administrators, 
and law clerks. Typically, participation in state programs is in the form of assistance in 
curriculum development, provision of text materials, and participation on the faculty 
by the program director and members of the educational committee. For example, dur- 
ing the grant period the project staff participated in the Kansas, Kentucky, and Wis- 
consin programs for the staffs of newly instituted intermediate appellate courts. 

Advice and cooperation are similarly provided to national education and training 
programs.The project staff has worked closely with the staff of the Institute for Court 
Management (ICM) in developing a curriculum for training appellate court administra- 
tors. On occasion, the project has co-sponsored special programs with ICM and in- 
cluded ICM staff and faculty in its curriculum. 

The program director indicated that communications are maintained with the Na- 
tional Judicial College. Furthermore, appellate judges who lack trial experience are en- 
couraged to attend the college's programs. 

Finally, the project staff intends to contribute time to curriculum development 
for the 1979 World Peace Through Law Conference. Assistance from the project has 
been requested in the past, by the World Peace Through Law Conference in Manila; 
hov^ver, staff time was not available to respond. 

OTHER AJC PROJECTS 

The Project on Continuing Appellate Education is one of many projects conduc- 
ted by the Appellate Judges' Conference. The activities and products of many of the 
conference's projects are mutually complementary, according to the project staff. The 
other projects of the conference are: 

Improvement of Appellate Practice 

■ Technology for the Courts 

m Multi-Agency Problems in CriminalJustice 

■ Implementation of Standards of Professional Discipline 
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Writing Standards of Professional Discipline 



• Improvement of Appellate Congestion 

■ Federal-State Delineation of Jurisdiction 

■ Development of LLM at the University of Virginia 

■ Fund for Judicial knprovement 

ALUMNI 



The project staff has not organized a formal group of former seminar participants. 
The rationale for this is that with the constant turnover of appellate court judges and 
staffs, there is little to be gained by a formal alumni group. However, the project staff 
has cited the development of an extensive communication network of appellate judges, 
clerks, and administrators as a direct result of participation in the seminars. 



Program and Related 
Costs 

The information that follows is a reflection of program and related costs, both 
actual and in-kind, for the May 1, 1976 to February 27, 1977 and March 11, 1977 to 
September 10, 1978 grant periods, 

FISCAL 

A summary of the project's recent fiscal history is displayed in the table entitled, 
"Summary of Revenues and Expenditures/' 

It is difficult to draw comparisons betvi/een these two grant periods due to the 
varying lengths of the grants. Therefore, analysis of the costs is in terms of percentage 
of the total funds rather than actual amounts. 

Revenue 

The proportion of the total project revenue contributed by LEAA discretionary 
funds for the 1976-77 grant period is 75 percent, compared to 81 percent for the 
1977-78 grant period. Project revenue is comprised totally of LEAA grant funds and 
contributions from private foundations. Additionally, a voluntary fee is collected from 
participants to cover the cost of other program activities which are not covered by 
grant funds- In 1979, for the first time, the project will also charge a tuition fee of 
$1 00 per participant. 



EKLC 



11-25 

112 



SUMMARY OF REVENUES AND EXPENDITURES 



May 1, 1976 
to 

February 27, 1977 



March 11,1977 
to 

September 10, 1978 



Revenues 

■ LEAA Funding 

■ Other Sponsors 

■ Tuition 

Total 

Expenditures 

e Salaries and Benefits 
« Instructional and Instructional Support 
(fees, travel, meeting facilities, etc.) 

■ Operation and Maintenance of Plant 
(utilities, furniture, etc.) 

Total 
Unexpended Balance 



$104,490 (74,5%) 
35,755 (25.4%) 
0 

$140,245 



$ 30,194 (21,5%) 

79,439 (56.6%) 

30,612 

$140,245 

0 



$163,121 (81.3%) 
37,511 (18.6%) 
0 

$200,832 



$ 63,932 (32%) 

95,617 (47.8%) 

40,205 (20.1%) 
$199,754 

$ 876 
(as of December 14, 1978) 



Expenditures 

Approximately 57 percent of the 1976-77 funds were used directly for delivery of 
the seminars. In 1977-78 the proportion was approximately 47 percent. This difference 
is related to the difference in funds expended for salaries and benefits in the two grant 
periods, approximately 22 percent in 1976-77 and approximately 32 percent in 1977- 
78. The percentage of funds used to cover facilities and general operations has re- 
mained relatively constant for the two grant periods, 20 percent and 21 percent re- 
spectively. 



In-Kind Contributions 



The value of contributed facilities and faculty time used in delivering educational 
services, as estimated by the project staff is extensive: a total of $80,880 in 1976-77 
and a total of $120,070 in 1977-78. * 

ACTIVITIES ANALYSIS 

The table entitled, "Programs and Related Costs," summarizes the staff effort and 
costs required for delivery of the various programs. 
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PROGRAMS AND RELATED COSTS (AJC) 



litMM ANU TcAil 


TOTAL NQ OF 
PROGRAMS 


AVG.NQ 

CTAPF/FAPIII TV 

REQUIREO (S/F) 


AV& PREP. 
HOURS S/F 


AVG. FACULTY 

lUCTDIIPTinUAl 

INalnULIIUNAL 
HOURS 


TOTAL NO. 
PARTICIPANTS 


AV& PARTICI- 
PANTS PER 
PROGRAM 


AVG. COST PER 
PROGRAM 


AVG. COST PER 

PROJECT 
PARTICIPANT 


late Judges' 
rence, 1976-77 


6 


2/10 


190/60 


25 


189 


32 


$4,700 


$146 


late Judges' 
rer>ce, 1977-78 


5 


2/10 


190/60 


25 


169 


34 


$5,600 


$164 


nal Conference of 
late Court Clerks, 
77 


2 


1/14 


Unknown 


Unknown 


132 


66 


$4,700 


$ 71 


ig of Appellate 
il Staff, 1976-77 


1 


0/7 


Unknown 


Unknown 


42 


Unknown 


$2,241 
(actual) 


$ 53 


ig of Appellate 
Jl Staff, 1977-78 


1 


0/8 


Unknown 


Unknown 


22 


Unknown 


$1,928 
(actual) 


$ 88 


llerks Model 
im* 


6 


0/0 


Not Applicable 


Not Applicable 


Not Applicable 


Unknown 


$1,627 


Not Applicable 



ogram delivered by states; only materials cost incurred by project. 

Ho 
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Appellate Judges ' Seminars 



The average staff time and effort required by the project for preparation of each 
seminar is 190 hours for two full-time staff. An average of ten faculty is used to deliver 
each seminar. Project staff estimates that the faculty spend a total of 60 hours prepar- 
ing topic outlines and materials for each seminar. Additionally, the faculty spend an av- 
erage of 25 hours each in delivery of the curriculum and informal exchange of informa- 
tion. 

The average number of participants at each seminar rose slightly in 1977-78, from 
32 in 1976-77 to 34 in 1977-78. This may be due to delivery of only five seminars in 
1977-78 compared to six in the previous grant period. 

The average cost of each seminar rose by $900, or 19 percent in 1977-78, from 
$4,700 per seminar in 1976-77 to $5,600 in 1977-78. At the same time, the average 
cost per participant rose from $146 to $164, or 12 percent. 

Since faculty do not receive compensation for their services, the portion of semi- 
nar costs attributed to faculty is for travel and subsistence. The travel and subsistence 
of faculty and staff consistently account for the largest portion of seminar costs, rang- 
ing from 36 percent to 65 percent for the two grant periods. The average cost for proj- 
ect staff salaries comprises the second largest cost of each seminar, an average of 32 
percent in 1976-77 and 28 percent in 1977-78. Finally, the materials portion of each 
seminar's cost in the two grant periods accounted for 23 percent and 22 percent, re- 
spectively. 

National Conference of Appellate Court Oerks 

Both programs for appellate court clerks took place in the 1976-77 grant period. 
The average total cost to the project is approximately $4,700, or an average of $71 per 
participant. Because all planning and preparation was done by the National Confer- 
ence, the project's contribution was primarily in the form of travel and subsistence for 
faculty, approximately 84 percent of the total costs. Other expenditures are attrib- 
utable to project staff time in connection with the programs. 

Meeting of Appellate Central Staff 

The project's contribution to these meetings is in the form of materials and cover- 
ing the cost of faculty and lecturers. An average of eight faculty are required to deliver 
these seminars to an average of 32 participants. The average cost of faculty travel and 
subsistence for each seminar is $1,123. The total cost of materials for each seminar var- 
ies with the number of participants in attendance; however, the average is $211, or 
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$6.60 per participant. The average total cost of each seminar incurred by the project is 



Law Oerks Model Program 



The total effort by project staff involves provision of the package program to 
state sponsors. Actual costs in providing the package to each state vary with the size of 
the program. However, the average cost is $1,627 per state. 



Management 

.... As stated earlier in this report, the project's overall management is the responsi- 
bility of the program director. All policies and procedures are governed jointly by the 
policies and procedures of the ABA and management procedures ijet by the program 
director and educational committee. 

Overall administrative policies and procedures are documented in two manuals. 
The first, ABA Policy and Procedures Handbook, outlines: 

Association Structure 

Sectiorjs and Committees 

l^embership 

Finances 

Prograrn Development and Funding 
Representation of the Association and Policy Formulation 
Meetings and Educational Programs 
Publications and Communications 
Index to ABA Policy on Legislative and National Issues 
Calendars 

The second, ABA Personnel Policies and Practices Manual, describes all categories of 
personnel and policies concerning the terms of employment with the ABA As a mem- 
cedtires °^ P''°j^^ subject to all of these policies and pro- 

The program development and management policies of the project are outlined in 
a conference management handbook. This document, developed by the program 
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director and chairman of the educational committee, basically contains timetables for 
program planning tasks and designates the individual or group responsible for each 
task. Additionally, a list of tasks to be performed by the project staff prior to every 
program has been developed by the program director. 

The project's budget development is the responsibility of the program director. 
The budget cycle proceeds as follows: 

■ The initial budget i$ developed by the project director. 

■ The budget is submitted to the Office of Resource Developmer)t, Fund 
for Public Education, for review and correction. 

■ 77?© grant application is completed by the program director and sub- 
mitted to LEA A. 

m LEAA responds to the grant application, 

■ The final application, in response to LEAA, is prepared by the program 
director. 

Financial management of project funds is handled, primarily, by the staff director 
for Grant Fund Administration of the Office of Resource Development, Fund for Pub- 
lic Education. This department provides the project with a regular accounting of ex- 
penditures. The program director initiates all transactions and expenditures; however, 
these are reviewed and approved by the financial department. 



Chattenges and Future 
Directions 

The ultimate goal of the Project on Continuing Appellate Education is to build its 
programs into an Institute for Appellate Justice. In order to accomplish this, several 
challenges and problems cited by the project, including the following, need to be re- 
solved. 

■ Entire groups of appellate Judges do not participate in the national 
seminars. In order for lateral communication to flourish among the 
judges, participation by as many appellate Judges as possible is neces- 
sary. This non-participation results in a void of information and per- 
spective. 

■ The project is understaffed. The program director says he spends a dis- 
proportionate amount of time on the appellate Judges' seminars, there- 
by neglecting the project's other programs. 

■ Funding for the project is inadequate. The project staff and educational 
committee devote a great deal of time to fund raising. This condition is 
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inhibiting to the program development process. 



The project staff sees these problems as msnor in comparison to the goals to be 
achieved- With expansion and improvement of its programs, the project staff hopes to 
realize its ultimate goal. 



Training Objectives/ 

Perceived Effectiveness 

Instructors at the Appellate Judges' Conference of the ABA feel they are very to 
fairly successful in achieving their priority learning objectives {Table 8B). They and the 
training participants regard the training as relevant to the needs of the courts repre- 
sented by the trainees. 

However, there are indications that the AJC is not as effective as the instructors 
believe (see Table 9B). For example, although one of the institute's stated objectives is 
to develop text material that will be used after the seminars, as well as during, 50 per- 
cent of the participants surveyed say they do not share training materials with others. 
Other indicators which should be watched in the future include the following: 



« 23 percent of the participant respondents do not wish to return to 
future conferences; 

■ On/y 55 percent of the participants say they attempted to make 
personal /professional changes as a result of things they learned at the 
conferences; and 

■ Only 45 percent tried to make organizational changes after attending 
training. 

There is also a gap between the perceptions of the instmctors and the participants 
where the soundess of the training design is concerned. As shown in Table lOB, parti- 
cipant ratings are quite low in seven of the 1 1 areas explored. In six of those seven 
areas, the instructors think they are doing much better than they are, at least in the 
opinion of their trainees. The areas of differences are: 

■ The degree to which an individuars needs are considered; 

• The degree to which expected outcomes are presented at the outset of 
the course; 

m The degree to which learning objectives are clear and succinct; 

• The degree to which learning objectives are demonstrable; 
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TABLE 8B 

INSTRUCTORS' PERCEPTIONS OF LEARNING OBJECTIVES (AJC) 



Psrceivgd Achievement 

AveregQ Category of 
Priority Rating Median 
Objective (T-High; 7-Low) Response 

1 . Update and increase partici- 1 .71 Very well 
pants' substantive knowledge. 

2. Improve participants' profi- 2.59 Very well 
ciency of their roles. 

3. Enhance participants' under- 3.06 Fairly Well/ 
standing of their roles. Very well 

4. Enable participants to intro- 3.53 Fairly Well 
duce new techniques and pro- 
cedures in their systems. 

5. Increase comnrtunication and 3.59 Very wet! 
consultation among profes- 
sional peers/colleagues. 

6. Enable participants to in- 4.53 Fairly Well 
f luence/promote change 

among other courts person- 
nel. 
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TABLE 9B 
MULTIPLIER EFFECTS (AJC) 



A$pect 



Typ© of 
Respondent 



Number/% 
Responding 



Frequency of Response 



Wish to return to 
institute. 

Perception of par- 
ticipants' desire to 
return to institute. 

Perception of par- 
ticipants' actual 
return to institute. 

Share institute 
materrais. 

Perception of par- 
ticipants' use of 
institute materials. 

Use institute materials. 

Attempt to make insti- 
tute-spurred per- 
sonal changes. 

Attempt to make insti- 
tute-spurred organi- 
zational changes. 

Recommend institute 
to others. 



ft 3L YES NO No response 

a 3k a JL 



Participant 

Instructor 

Instructor 
Participant 

Instructor 
Instructor 

Participant 

Participant 
Participant 



22 100 17 77 5 23 

20 95 20 100 - - 1 

16 76 16 100 - - 5 

22 100 11 50 11 50 

15 71 15 100 - - Q 

19 90 15 79 4 21 2 

22 100 12 55 10 45 

22 100 10 45 12 55 - 

22 100 19 86 3 14 
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TABLE 10B 
SOUNDNESS OF TRAINING DESIGN <AJC) 



Type of Number of IVtedian . Frequency of Response 

-aining Design Component Respondent Responding Response Not at all Somewhat Fairly well Very well Thoroughly No Response 



Degree based on 


Participant 


22 


Very well 




2 


3 


13 


4 




Drof fission's needs 




(100%) 






(9%) 


(13%) 


(59%) 


(18%) 




Instructor 


21 


Very well 


_ 




3 


11 


7 ■ 








(100%> 








(14%) 


(52%) 


(33%) 




Degree individual's 


Participant 


21 


Somewhat 


9 


3 


6 


3 




1 


needs are considered. 




(95%) 




(43%) 


(14%) 


(29%) 


(14%) 








Instructor 


20 


Very well 


1 


2 


3 


11 


3 


1 






(95%) 




(5%) 


(10%) 


(15%) 


(55%) 


(15%) 




Degree expected out- 


Participant 


18 


Not at all/ 


9 


3 


3 


2 . 


1 


4 


comes presented at 


Instructor 


(82%) 


Somewhat 


(50%) 


(17%) 


(17%) 


(11%) 


(6%) 




outset of course. 


17 


Fairly well 


1 


6 


2 


5 


3 


4 






(81%) 




(6%) 


(35%) 


(12%) 


(29%) 


(18%) 




Degree learning ob- 


Participant 


16 


Somewhat/ 


6 


2 


4 


3 


1 


6 


jectives clear and 




(73%) 


Fairly well 


(38%) 


(13%) 


(25%) 


(19%) 


(6%) 




succinct. 


Instructor 


19 


Very well 




3 


4 


8 


4 


2 






(90%) 






(16%) 


(21%) 


(42%) 


(21%) 




Degree objectives are 


Participant 


20 


Somewhat 


8 


4 


4 


4 




2 


demonstrable. 




(91%) 




(40%) 


(20%) 


(20%) 


(20%) 








Instructor 


19 


Very well 


1 


3 


5 


9 


1 


2 






(90%) 


15%) 


(16%) 


(26%) 


(47%) 


(5%) 




Degree training 


Participant 


19 


Not at all 


13 


1 


3 


2 




3 


provides opportunities 




(86%) 




(68%} 


(5%) 


(16%) 


(11%) 






to practice what is 


Instructor 


19 


Fairly well 


7 


2 


3 


5 


2 


2 


taught 




(90%) 




137%) 


111%) 


(16%) 


(26%) 


{11%) 
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TABLE 10B (Continued) 



Category 

Type of Number of Median Frequenc y of Response 

Training Design Component Respondent R^ponding Response Notata^^ 



informed of progress. 



materials, and activities 
cohesive. 



evaluation useful. 



g to provide input to 
faculty. 

11. Degree training up- 
graded and modified 
in relation to current 
state of the art. 



Participant 


14 


Not at all 


12 




(64%) 




(86%) 


Instructor 


16 


Not at all/ 


8 




(76%) 


Somewhat 


(50%) 


Participant 


22 


Very w/ell 


2 




(100%) 




(9%) 


Instructor 


19 


Very well 






(90%) 






Participant 


18 


Not at all/ 


9 


Instructor 


(82%) 


Somewhat 


(50%) 


17 


Very well 


3 




(81%) 




(18%) 


Participant 


22 


Very well 


1 




(100%) 




(5%) 


Instructor 


20 


Very well 






(95%) 





1 

(7%) 
3 

(19%) 



(11%) 



4 



1 

(7%) 
2 



5 

(23%) 
4 

(21%) 
4 

(22%) 



4 



2 



(13%) (13%) 



13 

(59%) 



2 

(11%) 
8 

(47%) 
9 



(18%) (18%) (41%) 



(6%) 



(9%) 
2 



(58%) (11%) 



2 



(18%) 



(18%) 



8 



1 3 8 

(B%) (15%) (40%) (40%) 



8 

5 



4 
4 
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The extent to w^ich training provides opportunities to practice what 
v\ms taught; 



The extent to which a trainee is informed of his/her progress; and 

The opportunity for and usefulness of participant feedback and evalua- 
tion. 



Training Setting, 

Techniques and Faculty 

t 

Both participants and instructors rate the Appellate Judges' Conference high on 
adequacy of the training settings {Table IIB). The instructors also give high ratings to 
management of the training events except for a low 50 percent when asked if program 
modification is based on feedback and systematic assessment {Table 12B). 

Participants and instructors agree that lectures and discussion groups represent the 
most used teaching techniques {Table 13B), and the participants say they find the 
lectures most useful. 

Participants rank instructors very high in the areas of substantive knowledge, 
practical experience, and teaching ability {Table 14B). They claim to have little if 
any professional contact with instructors after the conferences end, but the instructors 
claim somewhat more contact {Table I5B). Instructors also tend to be invited back. 
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TABLE 11B 

ADEQUACY OF THE TRAINING SETTING (AJC) 



Frequency of Response 



Aspect 


Type Of 
Respondent 


Number/% 
Responding 


Ye« 


No 


1. Simiianty of participants' 


Participant 


22 


21 


1 


roles/experience 








\OTbl 




Instructor 


21 


91 








(100%) 


\ 1 \J\JA>t 




2. Sufficiently small classes 


Participant 


22 


18 


4 














Instructor 


21 


1Q 


o 








\%J ITo/ 




3. Sufficiently long courses 


Participant 


22 


19 


3 








\OO70/ 


(14%) 




instructor 


20 

^w 


17 












(IOtoi 


4. Match between partici- 


Participant 


21 


19 


2 


Dant needs and instruc- 










tor expertise 


Instructor 


91 


on 


i 

1 






\ IUUa>| 




(5%) 


5. Availability of instruc- 


Participant 


22 


15 


7 


"tors nufsirfp of r«l»cc 




(IUUto) 


(68%) 


(32%) 




Instructor 


20 


17 


3 






(95%) 


(85%) 


(15%) 


6. Sufficient number of 


Particioant 


21 

mm 1 


1 o 




contact hours between 




(95%) 


\OD7o/ 




participants and in- 


Instructor 


20 


17 


3 


structors 




(95%) 


(85%) 


(15%) 


7. Reinforcement of be- 


Instructor 


17 


13 


4 


havior/skills 




(81%) 






8. Adequate support ser 


Participant 


21 


19 


2 


vices 




(95%) 


(91%) 


(9%) 




Instructor 


21 


20 


1 






(100%) 


(95%) 


(5%) 


9. Adequate physical ac- 


Participant 


22 


20 


2 


commodations 




(100%) 


(91%) 


(9%) 




Instructor 


20 


20 








(95%) 


(100%) 





1 

4 
1 
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TABLE 12B 

MANAGEMENT OF TRAINING EVENTS (AJC) 



Aspect 



1 • Clarity of lines of authority/ 
responsibility 

2. Adequacy of administrative 
procedures 

3. Instructors appraised of per- 
formance/expectations 

4. Instructors given orienta- 
tion/staff training 

5. Program modification based 
on assessment 



CTP Instructor Response 

Number 

Responding Yes No 



Jl Ji H ^ H 

20 95% 20 100^.. 

19 90% 19 100% ^ 

19 90% 17 90% 2 11% 

20 95% 18 90% 2 10% 
18 86% 9 50% 9 50% 
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TABLE 13B 
FREQUENCY OF METHODS USED (AJC) 





Method 


Median ResDonsa 






CTP Particioant 




1. 


Lectures 


Most of the time 


Most of the time 


2. 


Discussion groups 


Often/Most of 


Often 


3. 




the time 




Panel discussions 


Not at all 


Often 


A 


Case studies 


N/A 


Some 


5. 


Role playing, simulation 


Not at all 


Not at all 


6. 


Videotaping 


N/A 


Not at all 


7. 


Moving pictures 


N/A 


Not at all 


8. 


On-the-job training 


N/A 


Not at all 


9. 


Computer Assisted 


N/A 


Not at all 




Instruction 
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TABLE 14B 
PERCEIVED FACULTY COMPETENCE (AJC) 



Aspect 



Frequency of Response 



Category Totally Minimally 
Number of Median Unknow- Knowledge- 
Responding Response ledgeabte able 



Quite No 
Adequate Knowledgeable Expert Response 



Degree of their sub- 
stantive knowledge. 



22 Quite 
(100%) Knowledgeable 



2 

(9%) 



11 

(50%) 



9 

(41%) 



Extensiveness of 
their practical ex- 
perience. 



22 

(100%) 



Quite 

Knowledgeable 



(5%) 



4 

(18%) 



11 

(50%) 



6 

(27%) 



Teaching ability. 



22 

(100%) 



Quite 

Knowledgeable 



6 

(27%) 



Not at 
all 



Once or 
twice 



8 

(36%) 



8 

(36%) 



No 



Periodically Often Continually Response 



Number times 
called upon staff/ 
instructors since 
training for followup 
technical assistance. 



22 

(100%) 



Not at all 



19 

(86%) 



2 

(9%) 



(5%) 
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TABLE 15B 

FACULTY FOLLOWUP/CONT. RELATIONSHIP (AJC) 



Called upon by CTP 

Number of Times Particioante Invited bade tn lny tit..t» 

N % N % — 



None 5 29 

1-5 5 29 



1 5 
12 67 



6- 10 4 24 1 5 

11 -25 2 12 2 11 

Greater than 25 1 6 2 11 



INTEREST IN TEACHING AGAIN 
YES (%) NO (%) 
95 5 
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3. NATIONAL JUDICIAL COLLEGE 



The description which follows is based primarily on a visit to the National Judicial 
College in Reno, Nevada, on December 6, 7, and 8, 1978. The research on site in- 
cluded interviews with the Dean, Associate Dean, Staff Director of the Academic De- 
partment, Staff Director of the Finance Department, Staff Director of the Adminis- 
trative Department, Staff Director for Publications and Communications, Staff Direc- 
tor for Admission and Services/ and the Program Attorney, Documents and records of 
the College including records of participant evaluations and minutes of meetings of the 
Board of Trustees were made available to the study team. 

During the visit, the research team observed two one-hour training sessions. While 
attendance at these sessions was a useful exposure to the training operations of the 
College, it should be pointed out that observations of training were not systematically 
or comprehensively undertaken and thus do not constitute a major source of descrip- 
tion for this report. Similarly, the researchers had the opportunity to live in the same 
accommodations that participants typically use while at the College and to attend 
several training-related functions, such as a graduation ceremony and community ban- 
quet. These circumstances provided helpful vehicles for interaction with participants, 
staff, and faculty; the events themselves, however, do not fall within the purview of 
this report. 



History of 

the College 

On January 1, 1978, the National College of the State Judiciary became the 
National Judicial College. The adoption of a new name— and autonomous corporate 
status— marked nearly two decades of institutional development. 

In 1960, Justice Tom C. Clark of the United States Supreme Court provided the 
impetus for judicial educatiori of state trial judges when he organized and chaired the 
Joint Committee for the Effective Administration of Justice. From 1961 to 1963, 
under the Committee's aegis and with original funding from the W.K. Kellogg Foun- 
dation, a team of traveling judges was formed to conduct seminars in every state. The 
body of materials and experience that evolved during these seminars won the sponsor- 
ship of the Board of Governors of the American Bar Association (ABA), as well as con- 
tinued funding from Kellogg. Consequently, in 1963, tiie National College of State 
Trial Judges was founded to meet the continuing education needs of general juris- 
diction judges. The following year, under the initial leadership of Dean Ernest Friesen, 
the College held its first course, a one-month long session for 84 judges, at the Univer- 
sity of Colorado Law School jn Boulder. In 1965, the College received a grant of 
$210,000 per year for 10 years. A proviso of this grant was that the College move to a 
permanent home at the Reno campus of the University of Nevada. This relocation was 
effected under the stewardship of Judge Laurance M. Hyde. The College's location in 
Reno was further solidified when its physical plant, which contains administrative of- 
fices, a 46,000-volume library, the Tom C. Clark Auditorium, classrooms, and seminar 
rooms, was constructed with private foundation funds and dedicated in early 1972. 
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The College's ties with the University of Nevada are strong- The physical plant 
is deeded to the University, which supplies both utilities and janitorial/maintenance 
services. At the same time, the University's Board of Regents benefits both from the 
prestige of having judges study on the campus and from the guest lectureship and com- 
mittee membership contributions of Judicral College personnel. However, the relation- 
ship with the University is not a formal institutional affiliation. The College's primary 
formal affiliation lies with the ABA, which administers by contract its personnel fringe 
benefits and payroll actions and whose Board of Governors elects the College's 12 
policy-setting Directors— three of whom are nominated by the Board itself and nine by 
the Council of the ABA's Judicial Administration Division, In fact, until the College 
became a separate corporation in 1978 it was considered an activity of the ABA and 
its employees fell under the ABA's organizational auspices. 

Although one foundation's grants were curtailed in 1 973, another's funding was re- 
newed In 1975 for five years at $325,000 per year with another $75,000 per year al- 
located to the law library. The Law Enforcement Assistance Administration has 
granted funds for academic sessions since 1971; for 1978, LEAA grants totalled 
$210,000. As a result of these and several other funding sources, the 1978 budget 
totalled $1.5 million, 32 percent of which came from tuition fees and 55 percent from 
the foundation sources. This Increased foundation support has permitted a stable 
tuition since 1975. 

Since 1974, the beginning of its period of greatest growth, the College has been 
headed by Dean Ernst John Watts. He andthe Associate Dean, Colonel James Johnson, 
direct a full-time executive, professional, and classified staff of 35 in addition to a 
minimum of 10 temporary and part-time personnel. 



Goals and Objectives 
of the College 

From its inception, the College has sought to improve justice by increasing judge 
proficiency and the level of system modernization. In order to fulfill this overall goal, 
the College's resources have been directed towards providing judges with career judicial 
training and education. Its activities in this regard have been guided by four principal 
objectives: 

• To impart knowledge of the latest trends in the law and to update 
previous legal education. 

• To improve skills and techniques and upgrade the actual performance 
level in /udicial craftsmanship. 

• To establish values and standards in the performance of /udicial 
functions. 

• To increase judges' understanding of the importance of their responsi- 
bilities, their impact on society, and their place in the scheme of 
democratic government. 
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In support of these objectives, the College has developed a phased instructional 
approach which includes: a basic orientation, upon appointment or election, to the 
judicial function; clinical instruction, after 12 to 18 months on the bench. In a variety 
of function-related subjects; and periodic training— at least once every three to five 
years on a continuing basis. 



Ptograms 

History and Trends: The volume of program activities at the College has in- 
creased steadily and, particularly in recent years, rapidly. During the period from 
1966 to 1969, two sessions of one-month's duration each were held per year. Four 
one-week graduate sessions were inaugurated in 1970, and this figure was doubled the 
foSlowir g year In 1972, special court programs were added, as well as programs for 
limited jurisdiction and nonlawyer judges. A record high 35 residential programs were 
offered in 1978. Altogether, the College has issued more than 8,000 Certificates of 
Completion for resident sessions; well over half of these have been awarded to judges 
attending sessions since 1974. 

During its history, the College has conducted 271 extension programs for states 
in association with state supreme courts, judicial organizations, and law schools. Par- 
ticipants of these programs number over 17,000. In 1977, an extension program was 
conducted for the first time outside of the continental United States, in Guam. 

Additionally, since 1975 ten special and innovative programs, such as the United 
Nations' "Meeting of Experts on Expeditious and Equitable Handling of Criminal 
Cases," have been offered in conjunction with disciplines related to the judicial 
process. 

An analysis of programmatic trends (see "Five- Year Trend") shows a marked in- 
crease in the number of resident programs offered over the past 5 years and a steady de- 
crease in the number of extension programs conducted during the same period. How- 
ever, it should be noted that resident sessions are one to four weeks in duration while 
extension programs are less than a week and usually two days in duration. 



FIVE-YEAR TREND* 





1974 


1975 


1976 


1977 


1978 


Number of 
Programs Offered 


Resident 


17 


23 


28 


30 


36 


Extension 


28 


26 


18 


18 


14 


Number of 
Participants 


Resident 


906 


1071 


1123 1290 

i 


1226 


Extension 


1905 


2272 


1375 i 1188 1 902 

1 1 



*These figures do not include assistance to states, fecial and innovative programs. University of 
Nevada activities, and community activities. 
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Current Activities: Programs currently offered by the College are classified 
as Resident, Extension, and Special and Innovative. In addition, assistance and sup- 
port are available to all state and regional judicial programs. 

The Resident Programs are divided into three categories. Programs of the first 
category, the General Sessions, are designed to acquaint trial judges with major legal 
subjects and trial bench skills. The substance of the curriculum, \/vhich runs from two 
to four weeks,* depends on the scope of jurisdiction and prior legal training of the 
judge. The second category, Graduate Programs, offers an advanced level of instruction 
to the experienced judge who has typically completed a general course. Specialty 
Programs, the third resident category, treat particular legal areas, such as search and 
seizure, etc., in depth and cross lines of jurisdiction. 

Extension programs occur at the request of state supreme courts, judicial organi- 
zations, and law schools. These programs, which range in length from one day to one 
week, are held locally, statewide, or regionally, and their topics are geared to the needs 
of the particular judiciary sponsoring the program. 

Special and Innwative Programs have a principal theme or subject that is treated 
in a conference format by members of all branches of government and various judi- 
cially related disciplines. 

Additionally, special provisions are made for the families of resident session par- 
ticipants through the College's programs for spouses and young people. These are 
optional programs that include orientations, tours, social events, academic sessions, 
and the like. 

The "Summation of Current Programs," lists the specific major areas of activity 
in 1978 and the added (i.e., either new or repeated) course offerings for 1979. 

Needs Analysis: According to the Associate Dean and the Academic Director, 
the College uses a variety of means to assess constituency training needs and identify 
topics. 

Several features characterize this process: (1) Executive and professional staff 
members are often in the field, and they have frequent opportunities to meet judges 
and question them about their educative or training needs. In this regard, personal com- 
munications are central. (2) Meetings with the education committees of the National 
Conference of State Trial Judges, the National Conference of Special Court Judges, and 
the Conference of Administrative Law Judges of the American Bar Association are held 
annually to obtain feedback as to how the College is meeting their needs and to obtain 
their recommendations for new education and training programs. (3) The most 



Interviews with NJC participants at the 12 survey sites uncovered strong sentiment 
for shortening the four-week programs to three weelcs. 
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SUMMATION OF CURRENT PROGRAMS 



SESSIONS 


DURATION 


TARGET GROUP 


TOTAL 
NUMBER OF 
PARTICIPANTS 


OTHER OFFERINGS 
FOR 1979 


Myer: Spring 
Fall 


2 weeks 
2 weeks 


Special court (nonlawyer) judges 


513 
(inclusivfil 




1 Court Jurisdiction: 
Summar 
Fall 


2 weeks 
2 weeks 


Special court (lawyer) judges and 
graduates of nonlawyer session 






si Jurisdiction: 
Spring 
Summer 1 
Summer 2 
Fall 


3 weeks 

4 weeks 
4 weeks 
3 weeks 


Genera) jurisdiction judges; or full-time 
special court (lawyer) judges with broad 
jurisdiction; or graduates of Special 
Court Session 






istrative Law Procedures: 
Spring 
Fall 


1 week 
1 week 


Judges, hearing officers, commissioners, 
referees, and examiners of Federali state, 
regional, and local administrative 
agencies 
















ninat evidence-Spring 
lina) evidence-Fall 
1 litigation-Spring 
litigation-Fail 
ision-making process 
Ision-making skills 
^d techniques 
kince -Spring 
lence-Fall 
p and the trial 
f trends 

tencing/criminal law 
tencing felons-Spring 
encing felons-Fall 


1 week 
1 week 
1 week 
1 week 
1 week 

1 week 
1 week 

1 week 

2 weeks 
2 weeks 
2 weeks 
1 week 
1 week 


Judges who have completed Special 
Court Session; or General Jurisdiction 
Session; or Administrative Law Procedure 
Session; or Judges of General Jurisdiction 
with 5 years' service on bench; or full-time 
judges of Special Courts with broad juris- 
diction, lawyer-trained, with minimum 
S years' service on bench 


379 


■ administrative law-Spring and 
Fall (new! 

■ criminal law 

■ sentencing, correction and 
prisoners 

■ evidence (Sunvner) 



"0 

O 
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SUMMATION OF CURRENT PROGRAMS 



0 



SESSIONS 


DURATION 


TARGET GROUP 


TOTAL 
NUMBER OF 
PARTICIPANTS 


OTHER OFFERINGS 
FOR 1979 


SPECIALTY 

" administrative law 

advanced-Spring 
" administrative law 

advanced-Fall 

• alcohol and drugs 

■ court administration 

> equitable remedies 

■ evidence-Spring 

■ evidence-Fall 

■ family court proceedings 

> Dfobate court proceedings 

■ saiiitn and seizuri) 

• santencingmitdeineanants- 
Spring 

■ sentencing misdetneanants- 
Fill 

• traffic court 


All 
1 

Week 


For all judges, court commissioners, 
masters, referees, court administrators and 
otiier persons exercising jurisdiction related 
to tlie subject matter of the session; and 
administrative law advanced: for Federal 
APA administrative law judges and their 
state counterparts who have functioned as 
such for 3 years 


357 


' ■ state anti-ttust litigation 
(newl-Springand Fall 

■ administrative law: claims 
and benefits (new) 

■ minorities in the court 
(new) 

* administrative law-complex 
case 

• alcohol and dnjgs-Spring 

■ equitable relief and injunctions 

■ search and seizure-Spring 

■ court management-managing 
delay-Spring, Summer, Fall 


EXTENSION 
14 state and regional seminars 


3 days 
leverage) 


Administrative judges, court administrators, 
court clerks, trial judges, lower court judges 
of state and local courts 


912 




SPECIAL AND INNOVATIVE 
(1 program) 

Judicial Rule Making Workshop 
for State Supreme Court 
Justices 


2 days 


Chief justices/state supreme courts 


22 




SPOUSES/CHILDREN 


2 weeks 
(average) 




821 





^H^m doesnot k/utfe assistm mi suppo^ to stats and local judici^ education and training orsankations by mans ofsdyice, mtetials, and faco/y. 



immediate verification of accurate needs assessment is thought to be enrollment 
numbers, and it was pointed out that, on this basis, only one program ("Minority 
Perceptions") has teen inadequately assessed. (A) Post factum measures of satisfaction 
concerning specific offerings are routinely taken of all participants. Participants are 
asked to identify other areas which the College should address in its course offerings. 

Curriculum Development: The process by which curriculum is developed at 
the College is determined by whether a topic is traditional (i.e., existing) or new. 

In the case of existing programs, the emphases are on revision, refinerpent, and 
staying current with the law. An effort is made to avoid "reinventing the wheel/' but 
a program is generally conducted three times before it is fully consolidated. Judges' 
responses are taken routinely six months after each course is presented, and these re- 
sponses provide the basis for modifications. Concurrently, the Program Attorney, who 
is assigned by the Staff Director of the Academic Department to coordinate and de- 
velop the course, keeps abreast of germane literature and contacts selected judges 
about what should be taught in a given area. 

Once a new course is defined by the Dean on the basis of needs identification, the 
Staff Director of the Academic Department, who is responsible for overall coordina- 
tion of the curriculum, assigns the course to one of four Program Attorneys, who 
directs and monitors its further development. Both the Academic Director and the one 
Program Attorney who was interviewed used the anti-trust course as an illustration of 
the general curriculum development process. The development of this particular offer- 
ing included the following explicit steps: 

■ The Academic Director and the Program Attorney researched what 
other anti-trust programs had incorporated in the past. (The literature 
search is continued by the Program Attorney throughout the existence 
of the program and becomes, along with a developing, documented task 
analysis of a fudge who must make an anti-trust case mo\/e through the 
system, a main basis for the eventual selection of course materials. Ad- 
ditionally, the College has its own series of published outlines for 
/udicial reference on a variety of topics, developed by the research staff 
in coordination with key faculty members.) 

■ The Dean, Associate Dean, Academic Director, and Program Attorney 
contacted people in key areas of the country (i.e., those with experi- 
ence in state anti-trust issues), practitioners. Federal fudges, and 
members of the Department of Justice's Anti-Trust Division. 

■ Experts were identified through Department of Justice input and the 
College's own knowledge, and a meeting of such resources on the west 
coast was convened to discuss what should be taught regarding state 
anti-trust issues. 

■ The Program Attorney developed a tentative course outline. 

■ A second meeting was convened with key people on the east coast, 
who made suggestions for a ma/or revamping of the outline. 
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• A third meeting was held with prominent anti-trust people in the 
Midwost. 

• A new outline was developed which covered specific content, sched- 
uling, and instructional methodology. (According to the Academic 
Director, instructional approaches depend on suitability to the topic, 
and the repertoire of techniques at the College is eclectic. In genera/, 
the direction is said to be away from T-groups, gadgetry, and lectures 
sfftd to incorporate more field trips and role playing.) 

• The outline was sent for review and comment to all the people who 
had made input on the course's development 

Faculty Selection and Development: The teaching faculty of the College is 
chosen from prominent judges, law professors, attorney Sr and court administrators, 
as well as from professional fields as varied as philosophy, speech, and psychology, to 
name a few. Faculty members serve without compensation, a contribution in time that, 
in 1977, was cited to be in excess of $500,000. 

Two principal criteria guide the selection of faculty. The first is substantive 
knowledge and judiciai experience; the second is ability as a speaker. A third criterion, 
personality, was identified by one respondent as being connected yet somewhat dif- 
ferent from knowledge and speaking ability. This criterion was defined both as how 
well one "comes across'' and, alternatively, "how well one is accepted by the partici- 
pating judges/' 

Faculty are identified and i^aiected through at least one of three modes. The first 
mode is the College staff's firsthand knowledge. This would include faculty who haw 
conducted courses for the College in the past, graduates whose teaching potential was 
assessed while they were attending the College, experts who become known to staff in 
the context of course development discussions (such as those described for anti-trust), 
and persons whose ability has been proven in state programs. The second mode is 
referral, which would typically occur through the recommendation of a state judicial 
educator. The third mode would be a person's national prominence. 

At the research team's request, 22 resumes considered to be representative as a 
group of the College faculty universe were selected by the Associate Dean. (The 
numbers that emerge from ttiis particular group profile are only gross statistics, and 
the sample on which they are based is ton small and inexact for reliable generalization.) 
In this group: 

■ Sixty percent are National Judicial College graduates. 

• Inline years is the average number of years of faculty tenure (the low 
is four years, the high is 15 years, and the mode is seven years). 

m Thirty-three percent are state supreme court justices; 27% are circuit 
court Judges; 14% are district court judges; and the remainder include a 
county court judge, a United Nations administrator, and a law pro- 
fessor. (These figures are based on the current positions held by 
each.) 
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• The average number of years in the current job k 8.5, with the longest 
tenure being 23 years and the shortest one year (based on 19 of the 22 
resumes). 

■ One of the 22 faculty members is a woman. 

• No data are available from these particular resumes concerning age or 
race. 

For the implementation or delivery phase of instruction, four faculty develop- 
ment mechanisms are utilized: (1) Prior to the course, group discussion leader training 
is provided, including preparation on communication, group process, and such tech- 
nologies as the mock trial, role playing, and panel discussions. (2) The Program 
Artorney attends the instruction and meets with faculty members daily during the 
course itself in order to monitor the progress of both course and instructor, (3) Each 
Tuesday, the Dean, Executive, and Professional Staff meet with all faculty members, 
faculty advisors, and discussion leaders to obtain feedback and take necessary 
corrective actions. (4) Trainee evaluations provide post factum feedback with respect 
to the course content and the performance of the individual instructors. 

Composition and Utilization of Staff: The College's full-time staff consists of 36 
members: eight executive, five professional, and 23 classified staff. A summary profile 
of a staff is presented in the "Summary of Permanent Staff Characteristics." 



Effectiveness 

Marketing of Programs and Selection of Trainees: Alumni of the College are 
widely located. For example, virtually every jurisdiction visited in a recent sun/ey (see 
The National Manpower Survey of the CriminalJustice System, 1978) had sent judges 
to the College at least once. Such a nationwide distribution of graduates— nearly 
one-thind of the nation's judges-and faculty has an automatic impact on the marketing 
of programs, as does the publicity and visibility received by the Col!ege through 
frequent newspaper articles. These are reinforced by brochures and applications mailed 
to alt state, county, and municipal judges describing the content of sessions, dates, 
eligibility requirements, and costs. 

In addition, an advanced listing of resident sessions and costs is sent to state 
court administrators, chief justices of state supreme courts, presiding or chief judges of 
local courts, and officials of state and national judges' organizations to assist them in 
planning education programs for the subsequent year. Personal contact is maintained 
with a high percentage of these individuals nationwide. 

The application process is facilitated through the provision of detailed 
instructions according to state and the name and address of each state's contact person. 
The criteria by which participants are seJected are similarly delineated; these have been 
described in the "Summary of Current Programs" ('Target Group"). 
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SUMMARY OF PERMANENT STAFF CHARACTERISTICS 





Executive 

Ola 11 


Professional 
otatT 


Classified 
Staff 


Percentage with 
Baccalaureate 
u 89 rev 


75% 


100% 


43% 
(includes 1 as- 
sociate degree) 


Percentage with 
Graduate Degree 


50% 
(3 law, 1 MBA) 


100% 

(Slaw, 1 li- 
brary science) 


N/A 


Teaching Experience 
(Average Years) 


4 


0.6 


N/A 


Experience 
(Average Years) 


19 


10 


N/A 


Tenure at College 
(Average Years) 


7 


1.7 


2 


Average Annual 
Salary 


$32,475 


$19,739 


$10,577 


Average Age 


49.5 
(high: 65 
low: 38} 


37 
(high: 51 
low: 28) 


31 

(high: 59 
low: 19) 


Ratio 

Male/Female 


6/2 


6/0 


2/21 


* Racial/Ethnic Group 
Representation 


White 


White 


White 



•According to area rabor review (Reno SMSA 1978), the black community comprises only 1-64% of 
Reno's population. Only one black has applied for a job in three years. Additionally, College 
officials claim that, while they do solicit in the newspaper and single out minority groups, their 
solicitations have had no results because of the low minority population and tight labor market 
(an unemployment rate of 2.5%). 
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Monitoring and Assessing Programs, Faculty, and Trainees: The College uses 
two evaluation methods: in-session observations and written critiques by participants. 
At Resident, Extension, and Special Programs, instructional content, methodology, 
coverage, and style are constantly scrutinized and monitored by College observers. In 
order to improve teacher effectiveness and update content, deficiencies are called to 
the attention of faculty during breaks; the Tuesday meetings of faculty, faculty ad- 
visors, and discussion leaders are used to assess the reception and the impact of the 
instruction. 

At the end of each judicial seminar and Special Program, participants are re- 
quested to complete a critique form that assesses materials, instructor effectiveness, 
facilities, and personal benefits. For resident sessions, these critiques are completed: 
(1) prior to the session, in order to determine individual learning expectations; (2) at 
the end of the session, in order to suggest improvements and evaluate a broad range 
of instructional dimensions, including the importance of the subject, the value of the 
materials, and the effectiveness of the presentation; and (3) six months after the 
session, in order to gauge the extent to which expectations were fulfilled and training- 
influenced practices or procedures were initiated. The three forms from each partici- 
pant are correlated, and an overall, largely quantified assessment is then made of the 
session. 

For families accompanying judges, a "Spouses' Evaluation of Accommodations 
and Activities" questionnaire is administered. 

Outside Evaluations: Several evaluations of the College have been conducted 
by outside evaluators in recent years. No major problems concerning the content or 
the quality of the programs were uncovered, with caveats reserved for class size and 
"the unsatisfactory relationships between national and state training programs" 
(National Manpower Survey of the Criminal Justice System, 1978: 65). It should be 
noted, however, that the 1978 Report of the Judicial Education Study Group includes 
the College among the several national, in-resident judicial education institutes recom* 
mended for periodic mandatory attendance by all state trial judges. Further^ according 
to the National Center for State Courts Evaluation Report on "Package Institutions*' 
(1974: 22), "College outreach programs appear not to duplicate state and local pro- 
grams, simply because thev are offered only at the invitation of a state or local 
body/' 

An additional study, Heginbotham's description of interviews conducted with 
training participants (The Impact of the Court Improvement Training Package on 
Crime Rates and the Quality of Justice, 1975), identified several dimensions of change 
in knowledge and/or activity associated with their experience in the College's programs. 
Interviewees felt that, concretely, these resulted in fairer trials, greater precision and 
flexibility in sentencing, and more efficient court management. 



Other College Activities 

Research and Publications: Instructional and reference materials developed 
by and through the College are widely distributed. This is especially true in block grant 
states that cannot send courts personnel to resident sessions. 
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A wries of textbooks has been published on such varied court subjects as evidence, 
ethics, sentencing, appellate opinion preparation, inherent powers of the court, courts 
and the public, and courts and the news media, among others. Procedural pamphlets on 
the judicial role in plea bargaining and at the preliminary hearing have also been 
prepared. 

Currently, a national level bench book, the latest in a series of quick access study 
outlines for judges, is being developed. Other topics in the series include state appellate 
decisions, criminal law, judicial function, and judicial function for administrative law 
judges. 

Research and writing by staff, faculty, and participants are facilitated by the 
College's 46,000-volume law library, the largest law library in Nevada. 

Interrelations: Due to both its functions and administrative structure, the Col- 
lege interfaces with a variety of groups and organizations, including alumni, the general 
public, and a number of criminal justice-related organizations. 

While the College has no formal alumni organization, it does maintain contact 
with its graduates and faculty through periodic mailings, publications, social events, 
and field visits by staff members. Similarly, relations with the general public are 
directly served when staff members, particularly the Dean, address groups on issues re- 
lating to the judiciary and the courts in general. 

The College interacts with other organizations in varvinq degrees of constancy and 
formality. As was reported previously, ties with the University of Nevada/Reno are 
strong, although the organizational affiliation is not formal. Computer, purchasing, 
check disbursal, and maintenance services are provided by the University, and an inter- 
change of academic personnel occurs frequently. In additlon,the College houses a large 
portion of its resident session participants in the College Inn, which is owned by the 
Board of Regents of the University of Nevada. 

The College's formal affiliation with the ABA was noted earlier. In sum, the 
College is no longer an activity of the ABA, which nevertheless continues to administer 
a few financial and administrative actions and whose Board of Governors elects the 
College's 12 policy-making directors. 

On a professional level, there are periodic interchanges of ideas, materials, or 
faculty resources with other courts training facilities, such as the Institute for Court 
Management, and national professional organizations, such as the National Center for 
State Courts. By the same token, there appears to be an historical distance that is 
more than geographical between the College and the American Academy of Judicial 
Education, which in effect are in competition over the general and limited jurisdiction 
judicial populations. 

The College works closely with the National College of Juvenile Justice, which is 
housed in the same building. Similariy, the building is used as an education and training 
facility for the National Conference of Juvenile and Family Court Judges. In fact, the 
Judicial College Building was used for 92 programs involving 5,090 people other than 
Resident and Extension Program participants in 1978. 
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ThB College also interacts regulariy with a number of in-state training programs. 
Such interactions may involve consultants, joint planning and coonjination, or the 
provision of materials and instructors. 



As the recipient of funds from various sources enumerated previously, the Col- 
lege maintains ties with foundations, privata organizations, and one government 
organization, the Law Enforcement Assistance Administration. 

H'ogram and 

Related Costs 

The College's budget history during the period 1977 to 1978 (calendar year), in 
addition to available projections for 1979, is recapitulated in 'The Summary of Recent 
Budget History." 



Management and 

Administration 

Policy: Policy for the College is determined by its 12-member Boart! of Directors, 
which is elected for a three-year tenn by the ABA's Board of Governors. In addition to 
setting policy during its three scheduled meetings in October, March, and July, the 
directors have established four committees to manage the affairs of the College. These 
are the Executive Committee, the Committee on Administration, the Committee on 
Academic Affairs, and the Committee on Finance. Rules and procedures for Board 
operation are formal, and comprehensive minutes exist for each meeting. 

Corporate officers elected by the Board from among its members are: Frank J. 
Murray, Chairman; John A. Sutro, Vice-Chairman; L. Stanley Chauvin, Jr., Secretary; 
and Jordan J. Crouch, Treasurer Other directors include William H. Erickson, Roland J. 
Faricy, James R. Greenfield, William B. Lawless, Tim Murphy, Florence K. Murray, 
Ben F. Overton, and George H. Revelle. Staff members elected by the Board to posi- 
tions in the College are: Ernst J. Watts, Dean; James E. Johnson, Associate Dean; 
Michael Maloy, Assistant Secretary; and James F. Williams, Assistant Treasurer. 

Organization: The College is comprised of five operational components that 
are under the general direction of the Office of the Dean. These are the Academic, Ad- 
ministrative, and Finance Departments, the Department of Admissions and Services, 
and the Department of Publications and Communications. Another component is the 
law library, which is under the supervision of the Director of the Academic Depart- 
ment. Detailed statements of policy and procedure exist, as well as written descriptions 
of functional area responsibilities and individual jobs. 

A sophisticated management information system supports the organization of the 
College. A continuously updated manual transcript file is supplemented by an auto- 
mated system. Personnel files are entered in the ABA's computer; since 1971, the 
University of Nevada computer has process accounting transactions at no expense to 
the College. 
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SUMMARY OF RECENT BUDGET HISTORY 





1977 


1978 


1979 
(projectad) 


REVENUES 
« LEAA Discretionary 
Funding 

« Other Grants and 
lifOnxnouTions 

• Program income 

• Total 


$ 268,207 
(19£% of Total) 

586,507 
512,794 
$1,367,508 


$ 210,000 
(14.2% of Total) 

811,750 
455,000 
$1,476,750 


$ 210,006 
(11.7% of Total) 

950,000 
623,644 
$1,783,650 


COMPUTED VALUE OF CON- 
TRr>UTED FACULTY AND 
FACULTY ADVISOR TIME 


$560,000 


$603,450 


$677,970 


EXPENDITURES 

■ Personnel and Operating 

• Programs 

• Total 


770,127 
501,337 
$1,271,464 


940,350 
476,400 
$1,416,750 


1,053,250 
670,400 
$1,723,650 


TOTAL NUMBER OF 
TRAINING PARTICIPANTS 
(NJC PROGRAMS) 


3,236 


3 004 


3 400 


•AVERAGE COST PER PAR- 
TICIPANT (NJC PROGRAMS) 

NJC Programs Include: 

1 . Resident Sessions 

2. Extension Programs 

3. Special and Innovative 
Programs 

4. Spouses Program 


$393 
(For average of 
56 instructional 
hours for NJC 
programs) 


$472 
(For average of 
57 instructional 
hours for NJC 
programs) 


$507 
(For average of 
60 instaictional 
hours for NJC 
programs) 



•Figure arrived at by dividing total program com for year by total number of training participants for all cate- 
gories of NJC programs. 
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Budget and Fiscal Matters: The budget is determined by the Dean's Office 
with substantial assistance from the Finance Director on the basis of the programs the 
College wishes to conduct in a given calendar year. Historicni cost files are the principal 
grounds for forecasting each year's annual budget. 

Once it is approved by the Board of Directors and put into effect, the budget is 
subject to periodic analyses at the end of each semester's calendar. (DA general evalu- 
ation is conducted by the College at the end of May, August, and December on the 
basis of cost per student per day, and program reviev^^s are made month-by-month. 
(2) After the Finance Director reviews all major entries, the budget is entered into the 
University's computer 3000 ledger accounts, through which an internal review is con- 
ducted every semester. On a year-round basis, the computer runs a selected spot-review 
of 10%«12% of the College's 46,000 transactions, (3) Regular extemal audits are con- 
ducted by Ernst and Ernst or some other major firm. (4) Grantors may conduct 
separate audits. 

The College attempts to identify the most competitive sources for purchases and 
to buy in advance. The University, which gets State of Nevada rates, generally does the 
bidding, and University forms are used for normal purchases and ordering. Everything 
purcha^ has a seal on it for inventory and insurance purposes, and these are reviewed 
once a year. 

Payrolls are processed through the American Bar Association and the University. 
The University makes all other check payments, except in the case of a special 
checking account which is a $10,000 imprest fund principally utilized for reimburse- 
ments for faculty and faculty advisor expenses. In no instance are fewer than three 
people involved in issuing checks. 

') 

The College maintains a minimum amount of petty cash to give change for 
travelers checks and, through the University, has set up a check-cashing system for 
participants' use. 

Personnel: Personnel procedures at the College incorporate a combination of 
ABA and State of Nevada guidelines. Salaries for classified staff are consonant with 
state schedules in order to stay competitive with the local labor market. Executive and 
professional staff, who are drawn from a broader geographical pool, are not necessarily 
paid in accordance with state classifications. Fringe benefits, such as health and 
pension plans, are provided by the College on the basis of ABA criteria, while vacation 
and sick leave accruals are derived from state regulations. 

Executive and professional staff are recruited nationally through newspaper ad- 
vertisements and postings. Classified staff are recruited through newspapers and the 
services of seven affirmative action affiliate agencies. People with degrees are particu- 
larly sought. At the same time, it is policy at the College to promote from within as 
much as possible. Several of the staff interviewed remarked on the difficulty of at- 
tracting minority employees due to the low percentage of minority residents in Reno. 
At the time of the study team's visit, there was no minority repre^ntation on the 
staff. 
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Staff evaluations follow the state's system. Using the letter of hire, on which a 
checklist of job responsibilities is enumerated, a provisionary evaluation is conducted 
between the employee and his/her supervisor after 90 days on the job. Subsequent 
evaluations are made at the six -month point and then armualiy, when the supervisor is 
required to make a declaration of whether or not a merit increase will be recommended. 
All performance evaluations are reviewed by the appropriate department staff director, 
the Staff Director of the Administrative Department, and then the Dean for final ap- 
proval. 

Faculty: The main processes by which faculty are identified and recruited 
<l.e., from alumni, national reputation, in-state experience, and referral) were described 
earlier. Additionally, the College has a practice of using participant characteristics as a 
basis for identifying women and minority instructors and discussion leaders. 

Once a faculty member is selected, he or she receives a confimiatory letter that in- 
cludes provisions for compensation. (Generally, this means reimbursement. In the case 
of both participants and faculty, reimbursement is typically received before they 
depart for home.) Descriptions of previous curricula, a list of available teaching aids, 
and a schedule of lecture times accompanies this letter. Furthermore, new faculty are 
required to arrive at least one day early for orientation. 

Provisions for faculty supervision include monitoring and observation, generally 
by the Program Attorney. If a major problem develops. It is brought to the attention 
of the Academic Director or the Dean. Two evaluations are sent to each faculty 
member. One is a "report card" from participants; the other is an assessment of per- 
formance by the respective Program Attomey. 

Equipment: The College has maintenance contracts with IBM, Xerox, and the 
University. Additionally, operator checklists exist for every piece of equipment. 

Equipment is covered by a blanket insurance policy, and building security is pro- 
vided in response to Reno's recent upsurge in crime. 



Challenges and 

Future Directions 

Several events and trends will be pivotal in helping to determine the College's 
future courses of action. 

The College's projections are oased, in part, on its ability to obtain a portion of 
a $100 million corpus that is to be given away by 1980. Obviously, this will have a 
major bearing on the College's capacity to reduce core LEAA funding to zero by 
1981* In any case, the major foundation's funding on the current basis will end in 1980. 

College officials are enthusiastic about the possibility of developing a "University 
of Justice" which would, in effect, consolidate the separate activities of various courts 
training institutes, such as the Institute for Court Management and the Appellate 
Judges' Conference of the ABA* 
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The College is concerned with meeting the varied needs of a potential audience of 
approximately 23,000 judges, more than two-thirds of whom have not yet been served 
by its programs. One impediment to realizing this goal appears to be that many judges 
have not yet accepted the concept of judicial education. Additional difficulties lie in 
motivating leadership, i.e., chief justices, to support training and, of course, in funding 
it. 

Officials identified three foci for future activities. The first would emphasize joint 
training for judges and court administrators. The second effort would be directed at 
training both appellate and trial judges in order to increase understanding of the 
functions of each and the relationdiips between the two. The third focus would be the 
Federal courts, although this would have a tertian/ priority behind trial and appellate 
court training. 



Training Objectives/ 

Perceived Effectiveness 

Instructors at the National Judicial ColLje set as their priority teaching ob- 
jectives ones which are similar, if not identical, to those espoused by the College. 
Further, the instructors feel they are very successful in achieving those objectives 
{Table 8C), and both they and the participants regard the training as very relevant 
to the needs of the courts represented by the trainees. 

The instructors and the training participants rate NJC high in effectiveness, 
as judged by the multiplier effects of the training [Table 90. There is a low percen- 
tage of participants who have attempted to make organizational changes as a result 
of training, but 73 percent have attempted to make personal changes, mainly in the 
areas of evidence, jury instructions and general philosophy. The survey indicates that 
only 45 percent of the trainees recommend NJC to others. That response would tend 
to indicate a lack of need or opportunity to recommend NJC rather than any dis- 
satisfaction with it, especially since 90 percent of the participants surveyed say they 
wish to return for additional training. 

Participants and instructors alike speak highly of the soundness of the training 
design at NJC {Table lOO. The only negative is the degree to which the participant 
is kept informed of his/her progress during the course of the training. 



Training Setting, 

Techniques and Faculty 

Participants and instructors at NJC rate the college high on adequacy of the 
training setting {Table JIC), and the instructors are pleased with the nnianagement of 
the training events {Table J20. 

Lectures and discussion groups are identified by participants and faculty as the 
most used teaching method {Table JSC) and the trainees rank both fairly high as to 
usefulness. 
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TABLE 8C 

INSTRUCTORS' PERCEPTIONS OF LEARNING OBJECTIVES (NJC) 



Perceived Achievement 

Average Category of 
Priority Rating Median 
Objective (1-High;7-Low) Response 

1, Improve participants' pro- 2.58 Very well 

ficiency in their rotes, 

2- Enhance participants' under- 2-67 Very well 

standing of their roles, 

3. Update and increase partici- 2,82 Very wed 
pants' substantive knowledge. 

4. Enable participants' to intro- 3.35 Very well 
duce new techniques and pro- 
cedures in their systems. 

5. Increase communication and 3.96 Very well 
consultation among profes- 
sional peers/colleagues. 

6. Enable participants to in- 4.47 Fairly Well 
fluence/promote change 

among other courts person- 
net. 
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TABLE 9C 
MULTIPLIER EFFECTS (NJC) 



Aspect 



Type of 
Respondent 



Nuniber/% 
Responding 



Frequency of Response 



Wish to return to 
institute. 

Perception of par- 
ticipants' desire to 
return to institute. 

Perception of par- 
ticipants' actual 
return to institute. 

Share institute 
materials. 

Perception of par- 
ticipants' use of 
institute materials. 

Use institute materials. 

Attempt to make insti- 
tute-spurred per- 
sonal changes. 

Attempt to make insti- 
tute-spurred organi- 
zational changes. 

Recommend institute 
to others. 



N % YES NO No response 



Participant 114 99 103 90 11 



Instructor 47 

Participant 114 

Instructor 42 

Instructor 50 



Participant 114 
Participant 114 



10 



Instructor 48 94 48 100 - 



92 47 100 - 

99 106 93 8 7 

82 42 100 - 

98 44 88 6 12 



Participant 114 99 83 73 31 27 



99 52 46 62 54 



99 51 45 63 55 



4 
1 

9 
1 
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TABLE IOC 
SOUNDNESS OF TRAINING DESIGN (NJC) 



Catesory 



Frequency of Response 



1. Degree based on 
profession's needs. 



2. Degree individual's 
needs are considered. 



comes presented at 
outset of course. 



4. Degree learning ob- 
jectives clear and 
succinct. 



demonstrable. 



6. Degree training 
provides opportunities 
to practice what is 
taught. 



nespondent r 


lespondinc 


1 Response 


Not at all 


Somewhat Fairly well Very well 


Thoroughly No Response 


Participant 


114 


Very well 


mm 


5 


16 


43 


50 

144%) 


1 




(99%) 


very Wcii 




(4%) 


114%) 
4 


|007d) 




insuuuor 


51 




2 










(100%) 






(4%) 


(8%) 








Participant 


115 


Fairly well 


26 


21 


27 


28 


13 






(100%) 


Uprw tAioll 

vciy Well 


(23%) 


(18%) 


124%) 




111%! 
\\\7a\ 




Inctriiptnr 


50 




2 


12 


23 


13 


1 
1 




(98%) 






(4%) 


(24%) 




\L\)/\}f 


Participant 


106 


Very well 


10 


12 


26 


43 


15 


9 




(92%) 




(9%) 


(11%) 


(25%) 


(41%) 




In^triiPtnr 

IIIjUUwIUI 


48 


Vpn/ u/pII 
vciy well 


3 


4 


9 


20 


12 

1^ 






(94%) 




(6%) 


(8%) 


(19%) 


142%) 




Participant 


107 


Very well 


9 


9 


28 


47 


14 


8 


Instructor 


(93%) 




(8%) 


(8%) 


(26%) 


(44%) 


(13%) 


PA 

50 


Very well 


2 


3 


7 


23 


15 


1 




(98%) 




(4%) 


(6%) 


(14%) 


(46%) 


(30%) 


Participant 


109 


Fairly well 


9 


18 


29 


31 


22 


6 


Instructor 


(95%) 




(8%) 


(17%) 


(27%) 


(28%) 


(20%) 


50 


Very well 


1 


7 


12 


21 


9 


1 




(98%) 




(2%) 


(14%) 


(24%) 


(42%) 


(18%) 


Participant 


110 


Fairly well 


16 


21 


26 


30 


17 


5 


Instructor 


(96%) 


Very well 


(15%) 


(19%) 


(24%) 


(27%) 


(16%) 


49 


4 


6 


11 


16 


12 


2 




(96%) 




(8%) 


(12%) 


(22%) 


(33%) 


(25%) 


1 * i 1 



TABLE IOC (Continued) 



Type of Number of Median Frequency of Response 

Training Design Component Respondent Responding Response Not at all Somewhat Fairly well Very well Thoroughly No Response 



7. Degree participant 


Participant 


lUo 


noi di ail 


DZ 


1R 
tu 


l*t 


11 

1 1 




7 


informed of progress. 




(34%) 




(O/TOj 




1 10^1 


(in%i 

|IUA)| 






Instructor 


46 


Somewhat 


13 


11 


15 


6 


1 


5 






(90%) 




(28%) 


(24%) 


(33%) 


(13%) 


(2%) 




0. Degree oojeciives, 


ranicipant 


114 


\/nn/ Uioll 

Vcly Well 




4 


10 


65 


34 


1 

1 


materials, and activities 




(99%) 




(1%) 


(4%) 


(9%) 


(57%) 


(30%) 




cohesive. 


Instructor 


51 


Very well 


— 


3 


5 


28 


15 








11 (10%) 






16%) 


(10%) 

\ fvf 


(55%) 


(29%) 




9. Degree feedback/ 


Participant 


110 


Fairly well 


29 


17 


18 


31 


15 


5 


evaluation useful. 




(96%) 




(26%) 


(16%) 


(16%) 


(28%) 


(14%) 






Instructor 


50 


Very well 


3 


4 


4 


21 


18 


1 






(98%) 


(6%) 


, (8%) 


(8%) 


(42%) 


(36%) 




T 10. Degree of opportunity 


Participant 


114 


Very well 


5 


14 


21 


49 


25 


1 


i to provide input to 




(99%) 




(4%) 


(12%) 


(18%) 


(43%) 


(22%) 




faculty. 




















11. Degree training up- 


Instructor 


50 


Very well/ 


1 


2 


3 


19 


25 


1 


graded and modified 




(98%) 


Thoroughly 


(2%) 


(4%) 


(6%) 


(38%) 


(50%) 





in relation to current 
state of the art. 



TABLE nc 

ADEQUACY OF THE TRAINING SETTING (NJC) 



Type CM Numi>i>r/% 
Aspect Respondent Responding 

1. Sinnilarity of participants' Participant 114 
rolos/expenence (99%) 

Instructor 49 

(96%) 

2. Sufficiently small classes Participant 115 

(10<D%) 
Instructor 51 
(100%) 

3. Sufficiently long courses Participant 112 

(97%) 

Instructor 50 
(93%) 

4. Match between partici- Participant 113 
pant needs and rnstruc- (98%) 
tor expertise \ nstructor !5C 

5. Avat lability of instruc- Participant 112 
tors outside of class (97%; 

Instructor 51 
(100%) 

6. Sufficient nunr;ber of Participant 114 
contact hours between (99%) 
participants and in- Instructor 51 
structors (100%) 

7. Reinforcement of be- Instructor 46 
havior/skills (90%) 

8. Adequate support ser- Participant 112 
vices (97%, 

Instructor 51 
(100%) 

9. Adequate physical ac- Participant 1 1 5 
commodations ( 1 00%) 

Instructor 51 
(100%) 



Frequency of Response 

No 



Yes 


No 


Response 






1 


\0 1 70/ 






48 




2 


(98%) 


(2%) 




09 












HO 


5 


- 


(90%) 


(10%) 






Q 

o 


3 




\//b/ 




48 




1 


(96%) 


(4%) 




104 


Q 


2 


(92%) 






49 




1 


(98%) 


(2%) 


91 


21 


3 


(81%) 


(19%) 




47 


4 


— 


(92%) 


\O70/ 




IDA 




1 


(93%) 


(7%) 


39 


12 


- 


\/ /TOi 








5 


5 








111 


1 


3 


(99%) 


(1%) 




51 






(100%) 






109 


6 




(95%) 


(5%) 




51 






(100%) 







ERIC 



ll-44e 

1 5 ■/ 



TABLE 12C 

MANAGEMENT OF TRAINING EVENTS (NJC) 



CTP Instructor Re«ponsa 



Aspect 



1 . Clarity of lines of authority/ 
responsibility 

2. Adequacy of administrative 
procedures 

3. Instructors appraised of per- 
formance/expectations 

4. Instructors given orienta- 
tion/staff training 

5. Program modification based 
on assessment 



Number 
Responding 



N 



50 



49 



49 



51 



48 



% 



98% 



96% 



96% 



100% 



94% 



Yes 



N 



47 



48 



48 



48 



47 



94% 



98% 



98% 



94% 



98% 



No 



3 



J6 

6% 

2% 
2% 
6% 
2% 



ERIC 



ll-44f 



'0 6 



TABLE 13C 
FREQUENCY OF METHODS USED (NJC) 



M^^P^ Median Response 

CTP Participant CTP instructor 



1. 


Lectures 


Most of the time 


Often 


2. 


Discussion groups 


Often 


Often 


3. 


Panel discussions 


Not at all 


Some 


4. 


Case studies 


N/A 


Some 


5. 


Role playing, simulation 


Some 


Some 


6. 


Videotaping 


N/A 


Some 


7. 


Moving pictures 


N/A 


Some 


8. 


On-the-job training 


N/A 


Not at ail 


9. 


Computer Assisted 
Instruction 


N/A 


Not at all 
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NJC faculty are rated very highly by the participants, in terms of substantive 
knowledge, teaching ability, and extensiveness of their practical experience. (See 
Table 14C.) 



Participants indicate only rare contact with instructors following conclusion of 
training programs. Instructors indicate slightly more post-training contact (Table JSC), 
and show a marked tendency to return to NJC as instructors. 



Other Comments 

In response to a final open ended question regarding overall strengths and weak- 
nesses, 34 participant respondents cited only strengths, five only weaknesses, and a 
number of others offered both. High on the list of strengths was collegiality— getting 
together with other judges. Also high were the credentials of the faculty and the 
management of the training setting. The chief weaknesses cited were that some courses 
were too long (especially the four-week courses) and lectures were used to too great an 
extent. 
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TABLE 14C 
PERCEIVED FACULTY COMPETENCE |NJC) 



Frequency of Response 

Category Totally Minimally 
Number of Median Unknow- Knowledge- Quite No 

Aspect Responding Response ledgeable able Adequate Knowledgeable Expert Response 



Degree of their sub- 
stantive knowledge. 



Extensivenessof 
their practical ex- 
perience. 



Teaching ability. 



Number times 
called upon staff/ 
instructors since 
training for followup 
technical assistance. 



113 

(98%) Knowledgeable 



115 Expert 
1100%) 



113 



115 
(100%) 



Quite 

Knowledgeable 



1%) 



1%) 



Not at Once or 
all twice 



Not at all 86 20 
(75%) (17%) 



2 54 
12%) (48%) 



46 

(40%) 



25 61 

(22%) (54%) 



56 2 
(50%) 



59 

(51%) 



26 

(23%) 



No 



Periodically Often Continually Response 



7 2 

(6%) (2%) 



TABLE 15C 

FACULTY FOLLOWUP/CONT. RELATIONSHIP (NJC) 
Called upon by CTP 

Number of Times Participants Invited back to Institute 



N % N % 

None -8 17 1 2 

I -5 15 33 27 55 
6- 10 9 20 11 23 

II -25 6 13 10 20 
Greater than 25 8 17 0 0 



INTEREST IN TEACHING AGAIN 
YES (%) NO (%) 
96 4 



1 1 -45b 



4. AMERICAN ACADEMY OF JUDICIAL EDUCATION 



The American Academy of Judicial Education (AAJE), located in Washington, 
D.C., was selected as the pilot site visit, to sen/e as a training session for the project 
staff who would conduct future visits. The proximity of the Academy to the evalua- 
tor's offices also permitted additional data collection and discussions with the staff 
after the visit was completed. 

The visit also assisted in the refinement of the interview instruments and in 
sharpening the goals and agenda for each future site visit. 

The site visit to the AAJE was held on November 15 and 16, 1978. The kick-off 
meeting included all members of the project team and all administrative staff persons 
from AAJE. The various project teams interviewed appropriate personnel to complete 
the survey questionnaires. Major interviews were conducted with the Executive Direc- 
tor; Chief, Programs and Services Section; and Chief, Administrative Section, who pro- 
vided much of the resource materials. Almost all staff members were involved in some 
phase of the interview process, especially regarding the administrative procedure of the 
office and programs. 

Several subsequent visits and telephone calls were made to complete the data col- 
lection process and to verify information. A draft of the findings was submitted to 
AAJE on January 15 for confirmation of facts. 



History of Academy 

The American Academy of Judicial Education was founded in 1969 by the 
American Judges Association and the American Judicature Society to provide continu- 
ing education for judges of limited and special jurisdiction. The first LEAA^rant was 
awarded in 1970 to the University of Alabama for a national program, offered and 
operated by the Alabama Program of Continuing Legal Education. Another LEAA 
grant was awarded to the university for a national program in 1971. Mr. Douglas 
Lanford directed both of these programs, in his capacity as the full-time Director of 
the Alabama Program of Continuing Legal Education. He became full-time Director of 
the Academy in October 1971. The offices were moved to Washington in 1972 when 
the LEAA grant was awarded directly to the Academy. 

In addition to the Executive Director, the Academy is presently staffed by a 
Chief, Programs and Service Section; a Chief, Administrative Section; a Chief, Con- 
ference Section; a Chief, Financial Section; and three secretarial/clerical staff members. 
From 1977 through 1978, a project director, whose position was funded primarily by 
the Department of Transportation, supervised a national judicial conference. 
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The Board of Directors of the Academy consists of six members; four of these 
are judges of state courts. Meetings are held twice annually to determine general policy 
and direction of the Academy and to offer guidance to the Academy Director. 

In addition, a Curriculum Committee, consisting of five members, meets once a 
year to determine the substance and form of the special national conferences. 

The major source of funding for the Academy is LEAA. Some funds have been 
provided by the National institute on Alcohol and Alcohol Abuse and the National 
Endowment of Humanities. Approximately 50% of the funds are from LEAA and 50% 
from project income. In 1978, the consideration of LEAA as a technical assistance 
organization eliminated the need for cash match requirements. This year, however, 
LEAA has again required this financial match. Unlike two other major national training 
organizations (National College of Juvenile Justice and the National Judicial College), 
the Academy has no foundation funding. 

The American Judicature Society in 1977 withdrew its affiliation with the Acade- 
my, citing its inability to provide a cash match for sponsorship as its reason for termin- 
inating its support. 



Goals and Objectives 

of the Institution 

The Academy seeks to improve judicial performance by providing superior con- 
tinuing education programs for judges, designed to help increase their knowledge and 
skills and, in some instances, modify their attitudes. Based on thetr concept of "master 
nriodel curricula," the Academy offers a wide range of programs and services to bene- 
fit a judge in alt phases of a career— from pre-bench orientation sessions, to videotaped 
materials and speciality seminars that serve as reviews for more experienced tria! judges. 

A more specific goal of the Academy is to assist states in implementing in-state 
programming and services by developing model programs, by offering technical assis- 
tance (personnel and materials), and by exploring cooperative efforts between regions. 

Programs 

Trends: In the early years of the Academy, the national conferences dealt pri- 
marily with traditional topics^riminal law procedure, evidence, legal writing, and 
sentencing, for example. The Academy placed emphasis, during this time, on the de- 
velopment of innovative methods of presentation, such as the videotaped mock trial 
and videotaped courses of instruction. The focus of the first few years was also on pro- 
viding assistance to states through the design and presentation of in-state conferences. 

Since 1974, the Academy has emphasized the development of innovative pro- 
grams, such as the judicial writing program and the expansion of the National Video- 
tape Library. The trend during the past few years has been to present programs which 
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address speciality issues, as opposed to procedural issues and skills. Conferences in the 
last two years include one-week programs on Jurisprudence and the Humanities, the 
Uniform Commerical Code, and the Alcohol Offender. The 1979 programs continue 
this trend, with programs such as Law and Psychiatry and Fact Finding, Decision Mak- 
ing and Communication. 

The greatest priority of the Academy at this time is to increase the effectiveness 
of its services to state trial judges by expanding the resources of the National Video- 
tape Library and by developing model programs and other types of materials that can 
be replicated by state judicial education officers. 

Although in-state programming is decreasing, the Academy is placing heavy em- 
phasis on regional judicial education planning. During 1978, for example, the Academy 
initiated regional judicial education planning committees in seven regions. One of these 
regions. New England, has been utilizing Academy technical assistance for over six 
years in order to plan regional programs. It is the Academy's goal to initiate and sup- 
port these regional efforts because they tower travel costs, complement in-state pro- 
grams, and result in fewer scheduling problems. 

The number of national conferences has increased 100% since 1974— from 10 pro- 
grams during 1974 to 20 during 1978. During the same time period, the number of 
state conferences has been decreasing (see chart below), a result of the increasing num- 
ber of states that have individuals responsible for developing in-state conferences. 
Participation in the national conferences fluctuated somewhat during the past five 
years, but participation during 1978 reached a record number of 426. 



PROGRAM CHARACTERISTICS 1974-1978 





YEAR 


PROGRAMS 


1974 


1975 


1976 


1977 


1978 


No. National Conferences 


10 


12 


12 


18 


20 


No. Participants (National) 


346 


318 


399 


340 


426 


No. State Conferences 


34 


28 


24 


20 


12 


No. Participants (State) 


2,608 


2,377 


1,916 


1,331 


1.192 


Average No. Contact Days Per 
Conference 


2-1/2 days 


2-1/2 days 


2-1/2 days 


5 days 


5 days 



Current Programs: During 1978, the Academy conducted 20 national confer- 
ences. The agenda included: 



Trial Judges Academy. A two-week national educational program for newly elect- 
ed judges of courts of general and limited jurisdiction, which serves also as a valuable 
refresher for experienced legally trained judges. The goal of the course is to provide the 
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basic knowledge, skills, and attitPdes necessary to properly discharge judicial responsi- 
bilities, particularly in the trying of cases. The format for this course includes short 
lectures followed by small group discussion or videotape workshops. Among the sub- 
jects dealt with are search and seizure, plea bargaining, speedy trial, evidence, sentenc* 
ing, and ethical concerns of enforcing law. Two of these programs were held during the 
summer, at the University of Virginia and the University of Colorado Law Schools. 

Citizens Judges Academy. Two one-week sessions are held concurrently with the 
triai judges academies aimed to train the non-lawyer judge to competently handle 
misdemeanors coming before him. Two sessions were held in 1978. 

Specialty Academies. The specialty academies are designed to provide intensive 
study for experienced trial judges, using videotaped simulated trial scenes as the basic 
tea^;hing technique, with foilow-up small group discussions concerning the rulings pre- 
sr^nted. Each conference runs for 2-1/2 days. Two are offered during the same week to 
allow judges to take consecutive courses during one week. Six specialty academies are 
normally held each year. These programs can bo adapted by individual states for their 
own needs. 

Trial Judges Writing and Appellate Judges Writing Program. These programs aim 
to assist judges in the improvement of writing skills, especially in their opinion writing. 
The method of instruction is basically writing and re-writing, under expert tutelage. 
This Is supplemented by private tutorials, and a modest annount of lecturing and small 
group discussion. There is a follow-up by the student judges with the submission of 
writing samples on a monthly basis to faculty members for a period of 10 months after 
the program has ended. Four programs were held in 1978. These programs received ex- 
tremely high recognit'^^n and ratings during training participant interviews in the 12 
case study sites. 

Jury Trial Workshops. Programs for experienced law-trained judges which feature 
a complete simulated criminal trial. One week-long conference. 

Judge Trial Workshops. Designed for experienced law-trained judges with limited 
jurisdiction, using a simulated criminal trial. One week-long conference. 

The Academy assists states in conducting judicial education conferences by pro- 
viding assistance in their design, organization and administration. The Academy can 
design a model curriculum for individual nates on a wide range of subjects with lec- 
tures, videotape materials and recommended consultants offered as part of the pack- 
age. States may also request Academy staff and faculty for the presentations. Many of 
the programs are spinoffs from national programs, but are tailored to meet the specific 
needs of individual states. The "turn-key" concept allows for replication of Academy 
programs. 

Needs Analysis and Curriculum Development: There is no formal assessment 
undertaken in terms of curriculm development. The Executive Director identifies 
need on the basis of his 18 years of experience in the legal education profession, 
his review of other institution's brochures, and his awareness of current problems. 
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He also receives counsel from the Board of Directors and the Curriculum Committee 
which meet annually to discuss course content and methodology. 

The evaluations administered to participants after the national and state confer- 
ences are also used in developing curriculum. The student assesses faculty performance 
and subject presentation, both in terms of content and format and makes recommenda- 
tions for future program areas. These suggestions and observations are considered in 
planning courses for the following year. 

The Academy's review of literature of other national and state education organi- 
zations also provides information regarding topic selection. 

The Executive Director feels that this subjective approach to needs analysis is the 
only viable means at this time to accomplish this task since there is no research and 
development component within the program grant* He feels that the major strengths 
of the curriculum are its innovative courses, excellent methods of instruction, and 
comprehensive programming that meet a variety of needs of the states. 

He sees as a major problem AAJE's inability, because of time and budget con- 
straints, to plan and develop more intensively and to develop model materials and 
programs for their use and replication by otbir national and state organizations. 
The declining money base of LEAA, he believes, causes staff reduction— which In 
turn minimizes the time that could be spent on these types of projects. 

In addition, the Executive Director would like to see other developments: (1) 
gearing needs assessment to various categories of judges; (2) using experts in cur- 
riculum design, both academicians and practitioners; {3) staff review of legal peri- 
odicals for indication of needs; (4) more rigorous testing— pre-on-site and post- 
training exams; and (5) more input from the Curriculum Committee. 

No specific instructional objectives are set for each course. Most of the materials 
are developed by the Academy faculty members. 

Faculty Selection and Development: There are 62 faculty members listed in 
the Academy roster; 30 of these were utilized in the Academy's national programs dur- 
ing 1977. Most of the faculty are chosen by the Executive Director from his own ex- 
perience or contacts or recommendations by colleagues in the field. There are no de- 
finite criteria for selection, although expertise in a particular area and good teaching 
skills are prerequisites. 

Of the 62 faculty, four are female and one of the total number is an ethnic minor- 
ity. The average age of the faculty is 48 years, with ages ranging from 31 to 67 years. 
Twenty-seven of the faculty are judges, including one Chief Just-^^e; 22 are professors 
of law; six are professors of English and Humanities. Among th others are: one attor- 
ney, one writer, one professor of criminology, one psychiatrist, and one psychologist. 

Of the 30 used in last year's national programs, over half (17) were judges, six 
were professors of law, and four were professors of English/Humanities. 
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Faculty members, according to policy, receive $100 per day for the national con- 
ferences and $135 for state conferences. The average tenure of the faculty at the 
Academy is 4.2 years, with several members participating since its inception. 

There is no faculty development program. The faculty are given the course topic 
and materials before the sessions and meet their classes with little advice or supervision 
from the Academy staff. 

Permanent Staff: Douglas Lanford, the Executive Director of the Academy 
since its inception, is responsible for the policy and planning of the Academy, In terms 
of program development, he works chiefly with the judicial education personnel. The 
former Staff Attorney is now the Chief, Programs and Services Section (since Novem- 
ber 1977). There is no other staff attorney at present. The next person on the staff in 
order of administrative significance is the Chief, Administrative Section, who is 
basically involved with office management. 

Two positions which formerly existed— the Assistant Director for Administration 
and Finance and the Assistant Director for Programs and Services—were terminated 
when LEAA funds were decreased last year. The responsibilities of the Assistant Direc- 
tor of Programs and Services and those of Staff Attorney were combined into one posi- 
tion. The duties formerly discharged by the Assistant Director for Administration and 
Finance were allocated to two other staff members. The Chief, Conference Section, 
handles all the administrative responsibilities for national and state conferences. In 
former years, these responsibilities were divided between two separate conference 
coordinators. 

There are two secretaries and one administrative assistant who comprise the sup- 
port staff for the office. 

The staff is recruited through newspaper advertising, employment agencies and 
notices at colleges and laws schools. 



Effectiveness 

Marketing of Programs and Trainees: The Academy has a mailing list of 20,000 
judges who receive brochures and schedules of conferences for the year. The mailings 
are distributed nationwide. 

During the past five years, national programs have been for appellate, general, and 
limited jurisdiction judges. The 1978 jurisdictional breakdown was 40 percent limited jur- 
isdiction, 40 percent general jurisdiction, 13 percent non-lawyer judges, and 7 percent 
appellate jurisdiction. The non-lavi-^^er judges, of course, are all of limited jurisdiction. 
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The total percentage then, for limited jurisdiction participants would be over 50 per- 
cent. Some courses are exclusively for a particular category of judge such as the law- 
trained experienced judge, the newly elected judge or the non-lawyer judge. The 
Executive Director sees AAJE's target group as the state court judge. 

The only restrictions placed on attendance concern categories of judges who can- 
not attend courses designated for another group. For example, non*lawyer judges can- 
not attend programs for law-trained judgies. Four major categories which the Academy 
tries to target in specific courses are: new judges, non-lawyer judges, legally trained 
judges, and judges in a specific state. 

Monitoring and Assessments of Programs, Faculty and Trainees: The in-house 
evaluations consist of what the Director terms "happy sheets/' questionnaires dis- 
tributed to participants immediately after a conference. These evaluations have helped 
to determine tiie substance and format of the curriculum, selection of faculty^ the 
elimination of topics, and changed course schedules and the chronology of sessions. 

Outside Evaluations: The 1972 Heginbotham evaluation consisted of inter- 
views with judges who have participated in the Academy programs and used quotes 
from them as a primary source. The remarks are very genera! and descriptive, with little 
analysts or criticism. 

In 1975, the National Center for State Courts evaluated the Academy as one of 
the "package" institutions. This evaluation describes the program, staff, nature of the 
faculty and participants, but does not deal with management or curriculum develop- 
ment. The Executive Director states that he did not make much use of these evalua- 
tions. 



Other Institutional 
Activities 

Publication: The Academy publishes two books that assist judges in remain- 
ing current in their field. Recent Decisions, a digest of Supreme Court decisions pre- 
pared at the close of the court term, is published annually. The second publication is 
Constitutional Criminal Procedures, an up-to-date source book written by Professor 
Charles H. Whitebread of the University of Virginia. 

The Academy had published JudicicJ Education News, a bi-monthly newsletter, 
until late 1975, when it was discontinued because of insufficient funds and staffing 
prrblems. The Executive Director plans to re-institute the News, if funding permits. 
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The Academy has developed a National Videotape Library through which tapes 
(primarilv in the area of criminal law and evidence) can be purchased or rented. These 
tapes are accompanied by printed materials 5;uch as workbooks and quizzes to assist in 
the instructional process. 

The Data Bank of the Academy is a repository which is maintained to serve as a 
clearinghouse for judicial education materials. 

The major outreach to alumni is through the brochure mass mailing and occasion- 
al memoranda relating to boosting national conference attendance. Several brochures 
with application forms are printed each year, describing the offerings for the current 
year, as well as specific courses, publications, and services. 

A larger catalogue printed in 1973 serves as the major institutional policy source 
and outline of programs. 

External Relations: The Academy is aware it is providing some services sim- 
ilar to those offered by the National Judicial College (NJC) in Reno, Nevada, an- 
other LEAA-funded training institution. Quite early in its history, the Academy was 
emphasizing the training of judges of courts of limited jurisdiction at the national level. 
Since 1971 it has also made its services available to states for developing training pro- 
grams for appellate general, juvenile and limited jurisdiction judges. In the early 1970's 
national conferences were publicized to general jurisdiction judges, and with the ex- 
ception of the writing program in 1974, no programs were developed for appellate 
judges. No national programs for juvenile judges have ever been contemplated because 
of the existence of the National College of Juvenile Judges. 

According to the Academy, LEAA awarded the National Judicial College funds in 
1974 with the condition that those funds were to be expended only for general juris- 
diction training. The funds that year to the Academy were similarly conditioned, to be 
spent for training for limited jurisdiction judges. The Academy expressed a willingness 
to adhere to this arrangement, but the National Judicial College refused. The NJC 
secured a one-year exemption from the condition from the Administrator of LEAA, 
who later resigned, and the exemption was never terminated. 

There are various opinions regarding the competitive factors of NJC and the 
Academy. On the one hand, some staff at the Academy view the NJC as a competitor 
with an edge because it receives other private foundation monies, has a larger staff 
and budget, and classroom space at its institution. On the other hand, it is also felt 
that since both schools cannot meet the demands of the education of judges, it is es- 
sential that judges have a choice of training programs. In this sense, the competition is 
seen as a motivating factor for providing programs which are both current and ex- 
cellent in substance and training format. The competition factor is viewed by some 
as a deterrent to institutional parochialism. 
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The Academy also feels that while NJC enjoys economies of scale with fixed 
location, large staff and classroom facilities, the Academy has an advantage in more 
diverse teaching methodologies, more innovative courses, and more service to states. 
The Academy points out that judges would probably prefer to go to new locations 
for their courses, a benefit which the Academy allows, but which the NJC, with only 
a few off-site programs, does not share. 

The Academy mentions LEAA funding in its brochure of 1979 conferences, 
LEAA is also mentioned as a funding source in the Academy catalogue which attri- 
butes its success to the "continued backing" of LEAA and other funding sources. 



Program and 

Related Costs 

Budget History: The project budget for the last three years has remained some- 
what constant, both in total figures and in separate categories. The total budget for 
1976-77 was $437,899, with $250,109 coming from LEAA funding. In 1977-78, 
the amount was $504,399, with LEAA funding totaling $250,109. The 1978-79 bud- 
get was $437,238, with LEAA contributing $185,002. 



BUDGET HISTORY 



EXPENDITURES 


1976-1977 


1977-1978 


1978-1979 


REVENUES 








LEAA funding 


$250 J 09 


5250,109 


$185,002 


Other Income* 


$187,790 


S254,290 


$252,236 


EXPENDITURES 








Salaries & Wages 


180,510 


217,926 


216,801 


Fringe Benefits 


24,768 


31,928 


33,363 


Travel 


64,914 


51,542 


42.851 


Contractual Services 


76,155 


110,050 


74.683 


Supplies & Materials 


67,402 


92,953 


69,540 


Current Fixed Charges (rent, mortgage) 


24,150 






TOTAL BUDGET 


$437,899 


S504,399 


5437,238 



Other income may Indude the cash match from the American Judges Association and the Ameri- 
can Judicature Society, tuition, viaeotape ana publications income, and recovery of costs from 
State conferences. 
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Despite the decline in the level of funding, in 1978-79 the Academy has maintain- 
ed the same level of budget as two years earlier. In 1976. The $437,000 total budget re- 
mains the same, while LEAA funds decreased by approximately $65,000. In terms of 
the categories of expenditures, faculty salaries have decreased while administrative 
salaries have increased. There are several positions in this latter area, however, which 
have not been filled during the last grant year, and it is not clear to which functional 
category the money has been transferred. (See chart below.) 



FUNCTIONAL CATEGORIES OF EXPENDITURES 



CATEGORIES 


1976-1977 


1977-1978 


1 

1978-1979 


Instructional Salaries 


353,750 


$66,080 


$45,930 


Instructional Support 

(Support staff, nnaterials, etc.) 


154,721 


162,365 


120,262 


Institutional Support 

(Administrative salaries) 


229,428 


275,054 


271,348 



Cost Analysis: In compiling the statistics given below, the total cost of each 
conference was divided by the total number of participants for each, to derive the cost 
per student per conference. In the chart below, the average costs were calculated bv 
dividing the total costs per student by the total number of conferences. Staff salaries 
and indirect costs (such as rent, equipment, etc.) are not factored into this analysis 
because the bookkeeping methodology does not divide costs for national conferences. 
This causes the cost-per-student to be less than actual cost. 

a. National Conferences 

The Citizen Judges Academy, held in Charlottesville, Virginia, August 8-11^ 
1978, had the highest cost per student of all national conferences held during that 
year. The actual cost per student for this conference was $580.00. Twelve students 
participated for 33-1/4 instructional hours. Two full-time faculty and six part-time 
faculty participated and four AAJE staff members were present. 

A workshop on evidence issues, held January 23-25 in Coral Gables, Florida, had 
the lowest cost per student (S109) for all national conferences. Twenty-one students 
attended 16 instructional hours. Two faculty and one AAJE staff member participat- 
ed. 

The remaining national conference costs fall somewhere between the minimum 
and maximum numbers presented. 
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b. State Conferences 

The in*state conference on judicial procedures, held in St. Thomas, Virgin 
Islands on May 12 and 13, 1978, had the highest cost per student of all in-state con- 
ferences. Six judges (representing all of the judges of that court) attended a total of 12 
instructional hours, bringing the cost per student to $673. Two faculty and one AAJE 
staff member were present. This maximum is considerably above the average cost since 
the remainder of the state conferences are significantly lower. 

The lowest cost per student was $19.08 for the Mississippi Justice Court 
Judges Seminar in Jackson, Mississippi, February 19-21, 1978. Six faculty were 
present, along with 200 participants. Again, this number significantly differs from the 
average cost per student (see chart below). 



CONFERENCE STATISTICS 



Category 


Total No. 
Conferences 


Average Cost 
Per Conference 


Average No. 
Students 


Average Co»t Per 
Student Per Day 


Average Cost Per 
Student 


National 


18" 


$5,804 


22 


$61 


$264 


In -State 


10* 


$6,496 


90 


$34 


$72 



'indicates conferences with available data. 



These figures for both national and state conferences are based on costs provided 
by AAJE. Estimates for some staff salaries per hour were not available since their hours 
are not broken down according to each conference. The figures for state conferences 
include the indirect charges paid by tha state, a 21.63% rate established by LEAA 
LEAA funding will not pay for any indirect charges beyond this rate. 



Management 



The Executive Director is the chief planning and policy officer for the Academy. 
He delegates specific tasks to Section Chiefs, who must ultimately meet the approval 
of the Director. Thfi Executive Director prepares the annual LEAA grant application 
and plans other budgetary matters with the Section Chiefs. 

T:-,e Chief of Programs and Services Section is responsible for national and state 
programs, in terms of design, faculty recruitment, training materials and teaching 
methodology. He coordinates much of the day-to-day operations of the Academy, 
which includes coordinating services such as technical assistance activities, research 
and publications. 
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The Chiefs Conference Sec^'on, is concerned primarily with the physical arrange- 
ments of each conference— registration, facility, and communication with faculty 
and participants. The Coordinator also maintains post-conference records regarding 
attendance, finances, and evaluation. 

The Chief, Administrative Section, oversees the support staff and is in charge of 
filing, office procedures, and equipment maintenance. (See the organization chart 
below.) 

While there is no written goals statement of the Academy, there is a booklet pub- 
lished three years ago which does outline the objectives of the Academy. This describes 
in detail the various programs and activities undertaken and lists faculty and staff. 
While personnel have changed, the philosophy and program goals of the Academy have 
remained the same; therefore this publication can be used as a statement of objectives. 



ORGANIZATIONAL STRUCTURE 



Board 
of 

Directors 



Executive 
Director 



Chief, 
Conferences 



Chief, 
Administrative 
Servrse 



Chief, 
Programs & 
Services 




Secretary 



Clerk 



Secretary 



The Academy has written fiscal and accounting procedures which appear fairly 
complete m that a process for recordkeeping is adequately outlined, v he filinq 
manual and property records forms track equipment and furniture 
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The Academy maintains job descriptions for all staff (except the Executive Di- 
rector), and a procedures manual which documents office procedures and policies. The 
procedures manual has been in a state of development for several years and was 80% 
complete at the time of our visitation. There were no procedures for personnel evalua- 
tion, discipline or grievance issues, at least at that time. 

The conference review file sheets were complete, providrng for accurate records 
on each conference in terms of staff and faculty hours, salary, instructional hours, and 
participant profile. 



Challenges and 

Future Directions 

The Academy made several comments relating to future funding, programs, and 
needs, tn 197S, funding requests are being submitted to LEAA, the National Endow- 
ment for the Humanities, and the Department of Transportation. AAJE was seeking 
funds specifically for the education of non-law-tratned judges, ^ts request to LEAA 
is limited to $150,000, or $35,000 less than the grant last year. Mr. Lanford would 
like to see more money allowed for research and development— for example, faculty 
development programs. He would also recommend a higher allocation of LEAA money 
to organizations which do not receive foundation grants. 

Some long-term goals of the Academy include providing comprehensive service to 
ait judges, developing more model programs for the states (state-spt.rific packages) and 
moving toward regionalism in national conference programming. Several components 
of their master model curriculum which they would like to see developed include: 
(1) model non-lawyer judge curriculum and certification process; (2) model pre-bench 
orientation curriculum for non-lawyer and trial judges; (3) curriculum for mental health 
and the law; and (4) model curriculum for juvenile judge pre-bench education and long 
range in-state curriculum (in cooperation with the National College for Juvenile 
Judges). A general goal is the development and presentation of programs and services 
for those judges who most need them. This normally me^sns an emphasis on judges of 
lin.ited jurisdiction and lay judges. 

The Academy is in the process of developing a model curriculum on alcohol of- 
fenders and the judiciary, a tum-key program funded by NIAAA. 

They would also like to increase their videotape capacity, seek more audio-visual 
support for their programs and develop better systems for office management. 

The Academy feels this evaluation will be helpful to them in terms of the statis- 
tics compiled which will help in recordkeeping and in analysis of past conferences and 
in forecasting conferences. They presently do not cost out staff time by each program 
but plan to do so in the near future. 
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Some Academy staff feel that the Academy should focus its efforts on one type 
of program— national— while cutting down on state programs. It is felt there is a point 
of diminishing returns with numerous programs and small staff. 



Training Objectives/ 

Perceived Effectiveness 

The objectives of AAJE and those identified by the instructors surveyed are In ac- 
cord, and the instructors feel they are successful in achieving their two priority objec- 
tives but not as successful in pursuing the others {Table 8D). Nevertheless, both the 
participants and the instructors feel the training is fully or very relevant to the needs of 
the court systems from which the participants come. 

Participants and instructors also rank AAJE high generally in the multiplier ef- 
fects which speak to the effectiveness of the training. Two areas which should be 
noted concem the degree to which trainees share AAJE training materials with others 
(53%) and the degree to which participants attempt to make organizational changes 
based on training (28%). The latter point probably speaks more to the thrust of 
training at AAJE. (See Table 9D,) 

As for soundness of training design, AAJE is rated exceptionally high by partici- 
pants and instructors alike {Table lOD), 



Training Sei. 

Techniques and Faculty 

Participants and faculty alike express high satisfaction with the training settings at 
AAJE programs {Table IID), bi instructors indicate a degree of dissatislaction with 
the management of those trainii.g events {Table 12D), especially regarding orientation 
and training of faculty, and, to a lesser degree, the adequacy of administrative proce- 
dures. 

Trainees and instructors agree generally that discussion groups and lectures are the 
most prevalent teaching methods {Table 13D), and the participants find these equally 
useful. 

The instructors at AAJE receive the highest rating possible by participants, in the 
areas of substantive knowledge, practical experience, and teaching ability {Table 14D). 
The instructors tend to be invited back quite often to teach at AAJE programs (Table 
I5D). 



Other Comments 

In response to a general question regarding the overall strengths and weaknesses 
of AAJE, the participants were quite positive. Instructional content and expertise of 
the faculty vjere most commonly mentioned. Of the weaknesses mentioned, none 
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emerged as any kind of consensus from the participants. Instructors, however, did cite 
problems in the organization and management, specifically the absence of a full-time 
technical director or leader present at the seminars. 
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TABLE 8D 

INSTRUCTORS' PERCEPTIONS OF LEARNING OBJECTIVES (AAJE) 



Objective 



1. Improve participants' pro- 
ficiency in their roles. 

2. Update and increase partici- 
pants' substantive knowledge. 

3. Enhance participants' under- 
standing of their roles. 

4. Enable participants to intro- 
duce new techniques and 
procedures in their systems. 

5. Increase communication and 
consultation among profes- 
sional peers/colleagues. 

6. Enable participants to in- 
fluence/promote change 
among other courts person- 
nel. 



Perceived Achievement 

Average Category of 

Priority Rating Median 
(1-High; 7-Low) Response 



2.73 



Very well 



2.92 



Very well 



3.15 



Fairly Well 



3.56 



Fairly Well 



4.19 



Fairly Well 



4.56 



Fairly Well 
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TABLE 9D 
MULTIPLIER EFFECTS (AAJE) 



Aspect 



Type of 
Respondent 



Number/% 

Responding Frequency of Response 

J± % YES NO No response 
N % N % N 



Wish to return to 
institute. 

Perception of par- 
ticipants' desire to 
return to institute. 

Perception of par- 
ticipants' actual 
return to institute. 

Share institute 
materials. 

Perception of par- 
ticipants' use of 
ins'iituti^ materials. 

Use institute materials. 

Attempt to make insti- 
tute-spurred per- 
sonal changes. 

Attempt to make insti- 
tute-spurred organi- 
zational changes* 

Recommend institute 
to others. 



Participant 31 97 30 97 1 3 

Instructor 28 100 27 96 1 4 

Instructor 25 89 21 84 4 16 

Participant 32 100 17 53 15 47 

Instructor 22 79 22 100 - - 

Instructor 27 96 23 85 4 15 

Participant 32 100 23 72 9 28 

Participant 32 100 9 28 23 72 

Participant 32 100 27 84 5 16 



3 
1 

6 
1 
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TABLE 10D 
SOUNDNESSOF TRAINING DESIGN (AAJE) 



Type of Number of Median Frequency of Response 

Training Design Component Respondent Responding Response Not at all Somewhat Fairly well Very well Thoroughly No Response 



1. 


Degree based on 


Participant 


32 


Thoroughly 




1 


1 


9 


21 






profession's needs. 




(100%) 






(3%) 


(3%) 


(28%) 


(65%) 






insirucior 




very wei! 






-3 
J 


14 
It 


11 

1 1 


















111%! 




(Ml 




2. 


Degree individual's 


Participant 


31 


Very well 


1 


6 


3 


13 


8 


1 




needs are considered. 




(97%) 




(3%) 


(19%) 


(10%) 


(42%) 


(25%) 








Inctn irtnr 


97 

Li 


vciy Well 






u 


12 


t) 


1 

1 








\yD70| 








(44%) 


133%) 




3. 


Degree expected out- 


Partkipant 


28 


Very well 


4 


3 


4 


14 


3 


4 




comes presented at 




(88%) 




(14%) 


(11%) 


(14%) 


(50%) 


(11%) 


3 


7 


UUIScI 01 lOUlbc. 


Inctriirtnr 


25 


Pairlu iA/p|| 


1 

1 


4 


8 


7 


5 


A 

u 
0 






(89%) 




(4%) 


(16%) 


(32%) 


(28%) 


(20%) 




4 


Dpnrpp Iparnino ob* 

L/WUICw IvQIiIImUUM 


ParticiDant 


27 


Verv well 


1 


7 


3 


11 


5 


5 




jectives clear and 




(84%) 




(4%) 


(26%) 


(11%) 


(41%) 


(19%) 






succinct. 


Instructor 


28 


Very well 


1 


4 


5 


9 


9 










(100%) 




(4%) 


(14%) 


(18%) 


(32%) 


(32%) 




6. 


Degree objectives are 


Participant 


31 


Very well 


4 


2 


1 


11 


13 


1 




demonstrable. 




(97%) 




(13%) 


(7%) 


(3%) 


(36%) 


(42%) 








Instructor 


26 


Very well 




2 


6 


13 


5 


2 








(93%) 






(8%) 


(23%) 


(50%) 


(19%) 




6. 


Degree training 


Participant 


31 


Very well 


6 


3 


1 


7 


14 


1 




provides opportunities 




(97%) 


Very well/ 


(19%) 


(10%) 


(3%) 


(23%) 


(45%) 


2 




to practice what is 


Instructor 


26 


2 


1 


1 


9 


13 




taught 




(93%) 


Thoroughly 


(8%) 


(4%) 


(4%) 


(35%) 


(50%) 
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TABLE 10D (Continued) 



T«j«i-n«.- /» 1^^°^ of Median Frequency ui nnnuHw 

I Degree participant Participant 31 Fairly well 9 3 6 3 10 1 

informed of progrea (97%) (29%) (10%) (19%) 110%) (32%) 

Instructor 24 Very well 4 2 4 11 3 4 

(86%) (17%) (8%) (17%) (46%) (13%) 

8. Degree objectives, Participant 32 Thoroughly 1 - 1 12 18 - 
materials, and activities (100%) (3%) (3%) (38%) (56%) 
cohesive. instructor 28 Very well - 1 5 14 8 - 

(100%) (4%) (18%) (50%) (29%) 

9. Degree feedback/ Participant 32 Very well '5 3 4 7 13 - 
evaluation useful. (100%) (13%) (10%) (13%) (22%) (41%) 

Instructor 24 Very well 1 _ 3 n 9 4 

(86%) (4%) (13%) (46%) (38%) 



zIO. Degree of opportunity Participant 32 Very well - l 4 14 13 
i to provide input to (100%) (3%) (13%) (44%) (41%) 

g faculty. 

11. Degree training up- Instructor 26 Thoroughly - _ 2 9 15 
graded and modified (93%) (8%) (35%) (53%) 

in relation to current 
stats of the art. 
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TABLE 11D 

ADEQUACY OF THE TRAINING SETTING (AAJE) 



Frequency of Response 





noSfjQnaoni 


riBsponaing 


Vac 

Yes 


no 


iVO 

Response 


1. !• itinaritv of DarticiDi^nts' 


Participant 


32 


28 


4 




roies/experience 




(100%) 


(88%) 


(12%) 






Instructor 


27 


27 


— 


1 






(96%) 


(100%) 






2. Sufficiently small classes 


Participant 


32 


31 


1 








(100%) 


(97%) 


(3%) 






Instructor 


28 


28 


— 


— 




• 


(100%) 


(100%) 






3. Sufficientiy long courses 


Participant 


32 


27 


5 








(100%) 


(84%) 


(16%) 






Instructor 


28 


26 


2 


— 






(100%) 


(93%) 


(7%) 




4. Match between partici- 


Participant 


31 


31 




1 


pant needs and instruc- 




(97%) 


(100%) 






tor expertise 


Instructor 


26 


26 


— 


2 






(93%) 


■ (100%) 






5. Availability of instruc- 


Participant 


32 


30 


2 




tors outside of ciass 




(100%) 


(94%) 


(6%) 






Instructor 


26 


23 


3 


2 






(93%) 


(88%) 


(12%) 




6. Sufficient number of 


Participant 


32 


29 


3 


— 


contact hours between 




(100%) 


(91%) 


(9%) 




participants and in- 


Instructor 


28 


28 


— 


— 


structors 




(100%) 


(100%) 

^ ■ ^^^^ 






7. Reinforcement of be- 


Instructor 


26 


25 


1 


2 


havior/skills 




(93%) 


(96%) 


(4%) 




8. Adequate support ser- 


Participant 


30 


27 


3 


2 


vices 




(94%) 


(90%) 


(10%) 






Instructor 


27 


23 


4 


1 






(96%) 


(85%) 


(15%) 




9. Adequate physical ac- 


Participant 


32 


31 


1 




commodations 




(100%) 


(97%) 


(3%) 






instructor 


27 


27 




1 






(96%) 


(100%) 
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TABLE 12D 

MANAGEMENT OF TRAINING EVENTS (AAJE) 



CTP Instructor Response 



Aspect 



1. Clarity of lines of authority/ 
responsibility 

2. Adequacy of administrative 
procedures 

3. Instructors appraised of per 
formance/expectations 

4. Instructors given orienta- 
tion/staff training 

5. Program modification based 
on assessment 



Number 
Responding 



H 

26 

28 
26 
28 

22 



J6 

93% 



100% 
93% 

100% 
79% 



Yes 



23 



23 
23 
21 
21 



% 
88% 

82% 

88% 

75% 

95% 



3 
5 
3 
7 
1 



No 



_% 
12% 

18% 

12% 

25% 

5% 
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TABLE 13D 
FREQUENCY OF METHODS USED (AAJE) 





Method 


Median Response 






CTP Participant 


CTP Instructor 


1. 


Lectures 


vJTten 


Of tpn 

\J I LCI 1 


2. 


Discussion groups 


Ui en 


iviosx OT Lne Lime 


3. 


Panel discussions 


iNOt at al! 


bome 


4. 


Case studies 


N/A 


Often 


5. 


Role playing, simulation 


Not at alt 


Some 


6. 


Videotaping 


N/A 


Some/Often 


7. 


Moving pictures 


N/A 


Some 


8. 


On-the-job training 


N/A 


Not at all 


9. 


Computer Assisted 
Instruction 


N/A 


Not at all 
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TABLE 14D 
PERCEIVED FACULTY COMPETENCE (AAJE) 



Frequency of Response 

Category Totally Minimally 
Number of Median vJnknow- Knowledge- Quite No 

Aspect Responding Response ledgeable able Adequate Knowledgeable Expert Response 



Degree of their sub- 30 Expert 13 1 7 18 2 

stantive knowledge. (94%) (3%) (10%) (3%) {23%) (60%) 

Eytensiveness of 

their practical ex- 32 Expert - - 2 12 18 - 

perience. (100%) (6%) (33%) (55%) 

Teaching ability. 32 Expert - 1 1 12 18 - 

(100%! (3%) (3%) (38%) (56%) 

Not at Once or No 
all twice Periodically Often Continually Response 

Number times 

called upon staff/ 32 Not at all 20 5 4 1 2 

instructors since (100%) (63%) (16%) (13%) [:%) (6%) 

training for followup 
technical assistance. 



o 
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TABLE 15D 

FACULTY FOLLOWUP/CONT. RELATIONSHIP (AAJE) 



Called upon by CTP 

Number of Tlrnes Participants Invited back to Institute 

N % N % 



None 4 16 0 0 

1_5 13 52 13 50 

6- 10 3 12 7 27 

11 -25 3 12 2 8 

Greater than 25 2 8 4 16 



INTEREST IN TEACHING AGAIN 
YES (%) NO (%) 
96 4 
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5. INSTITUTE FOR COURT MANAGEMENT 



This report on the Institute for Court Management is based primarily on informa- 
tion obtained on a visit to the Institute In Denver, Colorado on December 11 and 12, 
1978. A two-person study team conducted interviews with the Executive Director, 
Director of Continuing Education, Director for Administration and Planning, Pro- 
gram Coordinator, and the accountant. The study team also worlced with administra- 
tive support staff who completed a number of data collection forms which docu- 
mented activities, staffing, and budget. 

All documents and records from the Institute were made available to the study 
team, including records of participant evaluations and minutes of the meetings of the 
Board of Trustees. Because of the voluminous nature of the documentation on pro- 
gram activities, these forms were completed by ICM staff after the study team departed 
and were later mailed for incorporation into this report. 



History of Institute 

.*, 'nstitute for Court Management (ICM) evolved in response to Chief Justice 

Warren E. Burger's 1969 speech to the American Bar Association (ABA) which cited 
a lack of professional management in the courts. As a result of this speech, a task force 
sponsored by the American Bar Association was formed to develop an educational 
program for court managers. With funding from the Johnson Foundation, the task 
force designed the Institute for Court Management and its initial educational curric- 
ulum. 

Joining the ABA in sponsoring the Institute were the American Judicature Soci- 
ety and the Institute of Judicial Administration. Each of these organizations nominates 
four members to the Institute's Board of Trustees. 

With a $750,000 grant from the Ford Foundation, the first class for the Court 
Executive Development Program (CEDP) began in June 1970 and was graduated in 
S^fthT?"* ^T?' ^'"'^^ ^^^2, classes in this core program have been funded, in part 
by the Law Enforcement Assistance Administration ( LEAA). 

Since 1970, approximately 305 individuals have completed the intensive Court 
Executive Development Program and have been certified as Fellows of the Institute 
for Court Management. Sixty-three percent of the ICM Fellows are in administrative 
positions in Federal, state, and trial courts; another twenty-five percent work in allied 
fields; and the balance are no longer clearly within the field or are deceased. 

Since its beginning, ICM has also developed additional educational programs to 
meet specific and ongoing needs, and has undertaken research activities in court 
management. Tfte Justice System Journal has been published by the Institute since 
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1974, serving as a forum for an exchange of ideas and information about the entire 
justice arena. 

Policy direction for the Institute is provided by the Board of Trustees which both 
initiates policy questions and responds to issues presented by the executive staff of 
the Institute. The current Executive Director of the Institute is Han/ey E. Solomon, 
an ICM graduate, who assumed leadership in 1974. The Executive Director heads a 
staff or five professionals and five technical and administrative support staff, who are 
located in ICM's administrative offices n downtown Denver, Colorado, and one 
professional who works on a permanent, part-time basis from his home in California. 

Current funding for the Institute is $582,000. Of that amount, 42% is generated 
by tuition and fees from its workshops and seminars, 39% by LEAA, 12% from contracts 
and consulting, and less than 10% from contributions and subscriptions. Programs par- 
tially funded by LEAA, such as the Court Executive Development Program, Strength- 
ening the Executive Component of the Court, and other seminars, are so noted in the 
Institute's announcements and annual reports. 



Goals and Objectives 
Of The Institute 

From the outset the Institute has sought to improve the management of the 
courts, thereby enhancing the quality of justice in courts and increasing popular 
satisfaction with the administration of justice. Specific objectives cited by ICM sre: 

a. To increase, store end disseminate reliable Icnowledge pertaining to 
effective and efficient court management, and to focus that knowledge 
on areas in need of improvement; 

b. To build acceptance of the court management function and develop the 
profession of court management; and 

c. To improve, through court management, popular satisfaction with the 
administration of justice. 

At its inception, the improvement of court management through improved business 
management practices was a primary goal of the Institute, Since 1974, the Institute 
has given increased attention to the relationship between judges and court administra* 
tors, and to the position of each in establishing goalf and policies in the court. 

Currently, the Institute ?s attempting to assess and rfcfonnulate its goals iind 
objectives in terms of today's needs in court management and the needs of professionals 
In court management. Accordingly, the Institute's Director of Administration and 
Planning has undertaken a Delphi process survey on rr)urt administration, involving 
judges, administrators, academics, and advcs^'Ses to assess needs and to consider 
alternatives in the Institute's mission. He has also prepared a detailed memorandum 
to the Program Development Committee of the Board of Trustees, documenting the 
Institute's history, its current target universe, and major policy alternatives. 
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Programs 



Currently the Institute's activities consist of educational programs, publications 
technical assistance, and court-related research. "Institute for Court Management Pro- 
grams" lists the specific major areas of activity and number of participants in 1978 



COURT EXECUTIVE DEVELOPMENT PROGRAM 

This program, the keystone of the ICM effort, is designed primarily for individuals 
who are now in, or in line for, key management positions in courts or closely related 
justice system agencies. Phase I, the Operational Side of Court Management, consists 
of five separate five-day workshops on these topics: Case Flow Management and Juror 
Utilization; Records, Systems, and Procedures; Information Processing Systems- 
Personnel Administration; Budget, Planning, and Financial Controls. Phase I! included 
;o"r-week residential seminar, a supervised study internship in a court system, and 

LmnLS'n^ n'^ ^"^^'^^^ '=«"°^ the Institute are awarded tS^^hc^ 

completing all requirements. 

ADVANCED AND CONTINUING EDUCATION PROGRAM 

fiwn^^?"" ^^l^ ^""^ conducted in various parts of the country, these three- to 
Sn^5 •^'^^o' °." """^ ^"P'"' ^ Courts and Community, Management for 

m"nS5^t°on. ^"^'"""^ -^""^ Management, and AppellateCourt a3- 

JUVENILE JUSTICE MANAGEMEIMIT 

TnnJcS^i*: ativanced seminars are offered in court management for juvenile courts. 
h.?S^* . ^ ^^^^^ seminars include: organization and structure of juvenile justice 
budget planning and management, caseflow and caseload management, personnel and 
probation management, and juvenile court law. 

STRENGTHENING THE EX ECUTIVE COMPONENT OF THE COURT 

This three-day seminar is an example of the Institute's efforts to develop inncn 
vative approaches and strategies. This LEAA-supported endeavor seeks to bring togetiier 
soMnr,°.ViIi?t« ^1 """^ ^^'T""'f rators from the same court to develop probiem- 
cSenw ^° relationships which continue after the 

TECHNICAL ASSIST ANCE: IN-STATE TRAINING PROGRAMS 

ICM provides workshops specifically tailored to the needs of states and other 
iniL^;"'°"w "contractual agreement with them. The contracting group! ^T'^ 

volved in advance m ^tting the goals and objectives of the training they are requesting 
thus gaming "ownership" of the program. In 1978, specialized training waTprov^dS 
in four jurisdictions in the States of Montana, Virginia, Pennsylvania and gZ^Is 
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Institute for Court Management Programs 



Activity 



Purpose/Target Group 



Duration 



Participants 



1. Court Executive Developrrient 
Program 

Phase I: Technology Work- 
shops (5) 

Phase II: Seminar 

internship 

Final Seminar 

2. Advanced and Continuing 
Education Program 



3. Juvenile Justice Management 



4. Strengthening the Executive 
Component of the Court 



5. Technical Assistance 
(ln*State Training 
Programs) 

6. The Justice System Journal 



7. Research: Misdemeanor 
Court Management Project 



Comprehensive court manage* 
ment training for individuals 
currently in, or in line for, 
key management positions 
in courts or closely related 
justice system agencies. 



For individuals with some 
training and experience 
seeking specialized knowledge 
in a particular field or an 
update in technical and ad- 
ministrative issues. 

Geared to organizational 
and managerial improve* 
ment of the juvenile court 
system. 

For teams of administrative 
judges and court administra* 
tors to improve problem- 
solving skills. 

Special program specifically 

designed and tailored to 

the needs of particular groups. 

A forum for the exchange of 
ideas and information on 
court-related issues. 



To identify shortcomings 
in misdemeanor court manage- 
ment and to propose innova- 
tions to meet certain needs. 



2 years 



5 days 
each 

28 days 



193 



24 



65 days {equivalent no. 

working days) 
8 days 32 



3-5 days 



6 days 



3 days 



3-5 days 



3 issues 
yearly 



ongoing 
through 

12/79 



479 



67 



192 



Approx. 
800 

Subscrip- 
tions 
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The Justice System Journal: In 1974 the Institute initiated this publication with 
the twofold purpose of providing a forum for the exchange of ideas and Information 
as well as an ingredient in the development of the profession of court manager. 

Research: Once a major activity of the Institute, research now constitutes a rela- 
tively small portion of ICM's time and resources. Current and recent research projects 
include the iVIisdemeanor Court Management Project and the West Virginia Jury Man- 
agement Study. {More recently ICM says it has taken steps to expand its research 
activities.) 

in addition to these activities, ICM staff respond to inquiries from graduates and 
the general public, serve as resource persons and speakers at conferences and symposia, 
serve on advisoi-y grotips, and do short-term technical assistance evaluations. 

Program Trends: While the major goal of the Institute, to improve the manage- 
ment of the courts, has not changed during recent years, the Institute has increased the 
number of workshops and seminars offered, expanded the variety of topics presented 
and experienced a broader range of positions, skills, and interests among the partici- 
pants. ^ 

The Increased number of workshops and the expanded variety of topics are 
related to the initiation in 1975 of the Advanced and Continuing Education Program 
and in 1977 of the Strengthening the Executive Component of the Court Program. 
About 50% of all the seminars conducted within these programs have been financially 
supported by LEAA funds. During this same period, the Institute began to conduct 
a number of educational programs based on technical assistance contracts with state 
and local court systems. 

The level of participants has shifted, in part, not only because of the new offerings, 
but also because the Institute recognized the need to contact a larger number of people 
who were associated with the many administrative processes taking place in courts. 
Originally the primary focus of the Institute's educational program centered on the 
senior administrative positions. By 1974 it became apparent that senior-level employees 
had to depend on middle management to implement and monitor processes essential 
to the functioning of the courts. It also became apparent that some of the middle 
management people were being promoted to more senior positions or had aspirations 
for such promotions. For these sorts of reasons, the Institute adjusted its perspective 
to en*:ompass this broader range of positions, skills, and interests among participants. 
The participants in the technical assistance activities, of course, are largely determined 
by the contracting party. 

Needs Assessment: The Institute utilizes a variety of mechanisms to assess the 
needs of the target audience: court administrators and related personnel. In 1976 a 
survey of the graduates of the Court Executive Development Program was used to 
identify topics in which additional training was needed. Current and recent studies 
and court cases also provide ICM with insights into the needs of court systems and the 
needs of administrators within the courts. Currently, the Institute is undertaking a 
large-scale Delphi exercise on the purposes of court administration, consulting with 
administrators, judges, academics, and advocates. Participant evaluations and staff 
monitoring of sessions also provide insights Into the needs of court administrators. 
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Additionally, tlie Board of Trustee's Program Development Committee, the staff, and 
unsolicited advice from former participants contribute to the Institute's identification 
of needs. 

Curriculum Development: The curriculum development process is closely depen- 
dent upon the formal and informal mechanisms for needs assessment. Once courses or 
topics are identified, learning objectives are formulated, faculty is selected, and presen- 
tations and exercises are developed. A major variation from this pattern is the in-state 
training provided upon request to states and local jurisdictions. In this instance, the re- 
questing agency determines the needs and participates in identifying the desired objec- 
tives. 

The underlying philosophy of training which guides the Institute's curriculum 
development is based on Malcolm Knowles' concept of "androgogy." Professor Knowles 
proposes an androgogy approach, i.e., treating participants as adults with a contribu- 
tion to make to the learning process rather than as children (pedagogy). Accordmgly, 
workshop - and seminars allow for interaction and exchange of ideas. As iCM's Direc- 
tor put it: "We're not trying to lay it on anybody." {The training workshop we ob- 
served utilized this .-approach with the result of total involvement by the trainees.) 

In developing the curriculum, the Institute's program directors negotiate and 
interface with the respective persons selected for the faculty. These discussions are 
based almost entirely upon a verbal understanding of the Institute's philosophy and 
approach to training; there are no written guidelines for these faculty to follow. 

Faculty Selection: Criteria for selecting faculty are: 

• Knowledge about the profession of court manager or of a particular 
topic area (together with the ability to relate the knowledge to the 
courts); 

• The ability to conceptualize problems and design exercises; 

m Willingness to '.vorf< with the institute in the design of a course; and _ 

• Good interpersonal sidlls. 

Increasingly, ICM is becoming aware of the need to identify professionals who have the 
ability to teach. Mechanisms for recruiting faculty are chiefly contacts through gradu- 
ates and former participants, other faculty members, and national organizations. 

Permanent Staff: The Institute's permanent staff consists of 1 1 members: six pro- 
fessional and five clerical or semi-professional. Permanent staff with responsibility for 
program design, development, and delivery are: 

■ The Executive Director is responsible for Phase I of the Court Execu- 
tive Development Program, oversees Phase II, and participates in Juve- 
nile Justice Management and in the Strengthening the Executive Com- 
ponent of the Court Program. 
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■ The Assistant Executive Director is responsible for the Juvenile Justice 
A/Ianagement Program, Probation and Managing Criminal Case seminars, 
and is the principal researcher for the Misdemeanor Court Management 
Research Project. 

■ 7776 Director of Continuing Education directs the Advanced and Con- 
tinuing Education Program, the In-State Programs, and also participatet 
in research on the National Study of Misdemeanor Courts. 

■ The Director of Administration and Planning serves as a faculty mem- 
ber for a variety of workshops and seminars. He is also responsible for 
mterfacing with the Board of Trustees and the Executive Director with 
respect to planning overall program policy. 

■ The Staff Associate for Court Management Programs coordinates Phase 
II of the Court Executive Development Program, is co-leader of the 
Juvenile Justice Mangement Program, edits The Justice System Journal, 
is a faculty member for some In-State and Advanced and Continuing 
Education Program seminars. 

A summary profile of the staff is provided in "Summary of Permanent Staff 
Characteristics." Roughly 13 percent of professional staff time and 77 percent of 
classified staff time 's spent on administration of the Institute. The typical professional 
spends 16 percent on curriculum development, 25 percent on seminar management 
and participation, 19 percent on research and publications, 14 percent on materials 
development, and 12 percent on marketing. {This information applies to the status of 
the permanent staff as it existed at the time of our visit to ICM and does not reflect 
the staff changes which have been made subsequently.) 



Effectiveness 

Marketing of programs and selection of Trainees: According to the National Man- 
power Survey of the Criminal Justice System (1978), there are 455 state and trial court 

on.Tln''^-"''^' ? 'V^^^ '"^ '«^el. On the average, the trial 

court administrators have five professional assistants and the state court administrators 
^rln ^" ^^^"9\°^ 12.6. Although there are no comparable figures for Federal courts 
ICM estimates that there are about 200 middle- and senior-level personnel in the Fed- 
era^ administrative structure. Additionally, ICM estimates that 300 staff from criminal 
justice agencies, universities, private consulting firms, and state appellate courts also 
tall within this middle- and senior-level category. Thus, approximately 3,600 middle- 
and senior-level managers constitute ICM's target population. Extending this universe 
to include clerks of court and judges' administrative assistants, an additional 2 000 per- 
sons potentially might be included. Given this rough estimate of the target universe 
ICM served 18% in 1977 and 18% in 1978. universe, 

it should be noted that because there are no recognized standards or qualifica- 
tions for many of the existing positions, there are no recognized, external incentives 
Tor these staff to seek out programs like those offered by the Institute. 
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Summary of Permanent Staff Characteristics 



Characteristics 


Professional Staff 
(N=6) 


Classified Staff 
{N-5) 


Percentage with 
Baccalaureate 


100% 


40% 


Percentage with 
Graduate Degree 


83% 




Teaching Experience 
(Average Years) 


8 




Justice Practice Experience 
(Average Years) 


12 


2.2 


Years with the Institute 
(Mean) 


5.5 


2.2 


Annual Salary 
(Mean) 


$31,720 


$12,242 


Age (Mean) 


42 


28 


Male/Female Ratio 


5/1 


1/4 


Racial/Ethnic 
Representation 


White 


White 



197 



Initially, the Institute trained a number of persons who had not had experience 
in the court systems. Increasingly, I CM has served those personnel with experience in 
the system, particularly those already employed in courts managment positions. 

ICM officials admit that the Institute has not excelled in marketing its services 
in recent years. 

While in the early 1970's there was a strong demand for the Court Executive 
Development Program (300 applicants for 30 slots), that demand has diminished 
owing to the decreased number of court management positions. At the same time, 
the Institute has adapted its marketing strategies to target those currently in the 
middle and lower administrative positions who will be assuming greater levels of 
responsibility in the coming years. 

According to ICM officials, the demand for training in the basic technology of 
courts management has leveled off. The advanced and continuing education and in- 
state educational programs are experiencing steady growth in demand. 

While the upcoming marketing strategy awaits the results of the overall needs 
assessment conducted by the Institute, the Institute currently maintains an open 
enrollment policy except for Phase II of the Court Executive Development Program. 
The Board of Trustees has requested that non-college graduates be no more than 10% 
of the enrollment in this latter program. 

Accordingly, criteria for selection of trainees, whiie basically "open," considers: 
(1) balance in the program according to geographic representation, type of courts, 
etc; (2) racial and sex balance; (3) representation from associated agencies; and (4) 
educational background. 

Monitoring and Assessment of Programs: The Institute employs several mecha- 
nisms to assess program effectiveness including: 

■ The amount of response to a course offering yvhidi is a ma/or index of 
tf^e value of the subject area to the prospective universe; and 

■ End'Of'Workshop evaluations which critique the value of the course 
and the performance of the instructors on an anonymous basis. 

With respect to end-of-workshop evaluations, the Institute, until recently, used a 
one-page overall program evaluation which utilized rating scales and open-ended 
questions. This was augmented by a one-page evaluation for each presentation which 
rated separately the faculty, subject matter, and written materials. The new evalua- 
tion form is a three*page summary which rates the general program, overall relevance 
of individual faculty and their topics, and elicits comments on improvements needed, 
materials, and logistics. The study team also noted a more detailed evaluation form 
which was used for the Workshop on Strengthening the Executive Component of 
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the Court. This instrument focused on attitudinai, knowledge, and behavioral out- 
comes with respect to each session and in light of the stated purpose of the session. 

Tabulations of the results of each of these evaluation forms are compiled by 
Institute staff and are used to assess topic relevance, foculvy performance, and future 
programming. 

Outside Evaluations: Two outside evalua Jons of the Institute and one tangential 
survey have been performed in recent years. The first, included in the National Center 
for State Courts Evaluation Report on Package Institutions (1974), noted the follow- 
ing points: 

■ /CM appears suited to meet the current needs of state court personnel 
in mid' and top-level management positions. 

■ The Institute should be prepared to offer its four- and five-week ses- 
sions at other times during the year (apart from summer). 

■ There is no apparent overlap between the offerings of university courses 
in court management and the Institute. I CM might, however, consider 
formaliziny its role as a provider of residency-type experience for 
graduates of court administration programs in universities, thus pro- 
viding a dimension of practical orientation. 

■ The Institute ought to be formally involved in the activities of the 
National Judicial College (formerly the National College of the State 
Judiciary) and the American Academy of Judicial Education. 

■ There ought to be a greater degree of cross-fertilization between ICM 
and the National College of Juvenile Justice. 

■ ICM ought to form a committee to develop programs for the upcoming 
year(s). 

The second evaluation was a follow-up case study on 14 ICM graduates who had 
been in senior court management positions for at least a year following completion 
of the course. This study, entitled Training for Court Executives: A Preliminary 
Assessment of An Institute for Court Management Program, was performed by Stanley 
J. Heginbotham and Jill A. Friedman for the National Center for State Courts in 1976. 
This brief report pointed up numerous concrete indications of changes in procedures, 
activities, and equipment in court systems which were directly attributable to ICM 
training. Impacts on the court system cited as a result of these changes included re- 
duction in time to select juries, and reduction in transcript backlogs and court costs. 

The National Manpower Survey of the Criminal Justice System (1978) recom- 
mended that tCM: 

■ Put increased emphasis on certain managerial skills, notably in the 
techniques for program review and evaluation; and 

■ Expand regional training services on more advanced management topics. 
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Other Institute Activities 
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External Relations: The Institute regularly works m cooperation with a variety of 
agencies and organizations. ICM's sponsorship by three organizations, the American 
Bar Association, the American Judicature Society, and the Institute of Judicial Admin- 
istration, has linked the Institute with these groups since its inception. 

In its research activities the Institute has co-ventured with the American Judica- 
ture Society in the Misdemeanor Court Management Project. 

In developing its courses, the Institute has utilized the resources of groups such 
as the National Center for State Courts and the National Pretrial Resource Center for 
substantive inputs and referrals to resource persons. 

The Juvenile Justice Management Program has been co-sponsored since its incep- 
tion by the National Council on Crime and Delinquency and tie Institute of Judicial 
Administration. 

Two years ago, the Institute conducted the education portion of the National 
Association of Court Administrators' national convention. 

Together with the University of Denver Law School. ICM conducted a 37-person 
conference on court administration for the Conference of State Court Administrators. 
The conference was jointly planned and jointly conducted. 

Workshops and seminars conducted for states and local jurisdictions are in effect 
co-ventures with the requesting parties inasmuch as they are jointly planned and spon- 
sored. 

In the past, ICM has discussed and planned co-ventured activity on court manage- 
ment with the National Judicial College (NJC). According to ICM officials, the co- 
venture never materialized because of an impasse over financial arrangements. ICM 
«aff anticipate, however, the possibility of joint activity in the near future with NJC. 
At the very least, ICM hopes to coordinate the presentation of courses dealing witti 
aspects of court management. 

Alumni Development: With approximately 305 Fellows of the Institute and hun- 
dreds of other former participants currently working in Federal, state, and trial court 
systems and in allied fields, the Institute has a valuable resource to draw upon. Some 
graduates senfe as instructors for current programs; others provide advice and referrals 
for coritent or instructors for the programs. The Institute has an Advisory Council of 
Alumni Fellows which serves as a formal link to its graduates. ICM has surveyed Its 
graduates to determine current needs and to document examples of applications of -the 
results of its training programs. Thus, the alumni are a critical resource for needs assess- 
ment and for marketing information. Additionally, graduates will contact the Institute 
to give or to obtain advice, or to make a financial contribution. 

The Institute has recently initiated a new program activity designed especially for 
graduates. These Renewal Seminars will be held approximately three times per year, 



1-69 



4'\ . 



sometimes in connection with annual professional meetings such as the National 
Association of Court Administration and the National Association of Trial Court Ad- 
ministrators. 



Program and Related Costs 

The Institute's overall revenues and expenditures have remained fairly constant 
over the past several years-at approximately $580,000 per year. "I CM Budgets, 1S76- 
1978" displays the budgets by major line items for the years 1976-1978. Only in 1977 
is there a significant variation— an unexpended balance of $45,905. The figures also 
indicate that the percentage of budget allocated to salaries and fringes has increased 
steadily from 42 percent in 1976 to 48 percent in 1978, whereas the amount spent 
on contractual services declined from 18 percent to 1 1 percent during the same period. 

The activities undertaken by the Institute over the past two years are summarized 
in "ICM Activities Analysis/' A comparison of the 1977 and 1978 activities indicates 
that the cost per participant has risen from $424 to $462. This increase appears to be 
related to an expansion of the more expensive course on Strengthening the Executive 
Component of the Court, and generally increased costs associated with the Court 
Executive Development Program, Advanced and Cortinuing Education, and In-State 
Training. With the exception of the Phase II of the Court Executive Development Pro- 
gram and the Strengthening the Executive Component of the Court Program, I CM 
seminars generally cost between $60 and $96 per participant day, excluding partici- 
pant travel. 



Management 

Policy Planning: Program and administrative policy for the Institute are set by its 
12-member Board of Trustees which is appointed by the American Bar Association, the 
American Judicature Society, and the Institute of Judicial Administration. The Board 
has an Executive Committee, a Program Development Committee, and a Finance Com- 
mittee. Since 1974, the Board, unti^T Chairman Earl F. Morris, has directed the execu- 
tive staff to propose program alternatives, budgets, and salaries of staff over $25,000 
per year. 3ased on the staff proposals, the Board deliberates and enacts policy. The 
Board exercises strict control over ICM policy, particularly over the budget. Minutes of 
the Board meetings are carefully kept, and reflect thorough fact finding and deliberate 
decisions. The Board not only reacts to staff proposals, but also initiates questions in- 
volving Institute objectives and policy. 

The Budget Process: In early June, the Institute staff prepares a tentative program 
for the following calendar year. This program is adjusted and refined during the ensu- 
ing weeks. The staff requirements for this program are then mapped out allowing 230 
days for a person-year. Tentative budgets for each program area are prepared and the 
staff reviews them, negotiates and finally translates the resulting package into a pro- 
posed budget. The budget is presented to the Board and its Finance Comminee in 
November. The Institute's Director for Administration and Planning then meets with 
the Finance Committee to go over the proposed budget and adjustments may be made. 
The Board of Trustees meets in December to review, adjust (if necessary), and adopt 
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ICM Budgets, 1976-1978 



BUDGET 


1 9/ W 


1Q77 


l9/0 


1. Revenue 








Tuition/Fees 

L*cr^>\ mnvJiny 
Contracts and Consulting 
Contributions 
Subscriptions and Other 


$231,329 

125,943 
20,380 
23,756 


$266,175 

1 99, J49 

71,822 
28,740 
21,740 


$241,600 
227,000 
72,100 
20,000 
22,400 


i Total 


$58^, So' 


$587,826 


$583,100 


2. Expenditures 








Salaries, Wages 

Direct 

Indirect 
Fringe Benefits 
Communications 
1 ravei anu oUDSisience 

Direct 

MUm 1 n IS L raXI Vi; 

Contractual Services 
Direct 

Administrative 
Supplies and Materials 
Current Fixed Charges 
(e.g., rent) 


$175,049 
38,691 
25,913 
12,327 

78,413 

93,013 
6,998 
50,283 
76,288 


$161,332 
42,918 
37,936 
11,044 

79,036 

o ceo 

71,172 
7,300 
43,447 
81,178 


$184,686 
39,976 
43,809 
11,000 

84,900 
8,054 

57,550 
6,750 
42,000 
82,975 


Total Expenditures 


566,260 


541,921 


562,300 


Unexpended Balance 


$ 18,607 


$ 45,905 


$ 20,800 



ICM Activities Analysis, 1978 



Activity (IM) 


Number of 
Participants 


Total 
Costs* 


Cost Per 
Participant 


Cost Per 
Participant Day* 


Court Executive 
Development 










1 Phase 1 (5) 
Phase 1 1 


193 
56 


$ 65,451 
119,070 


$ 339 
2,126 


$ 68 
NA»» 


' Juvenile Justice 
Manage- 
ment (3) 


67 


28,852 


430 


86 


Strengthening 
the Executive 
Component 
of the 

Court (2) 


40 


48,748 


1,218 


406 


Advanced and 
Continuing 
Education 
Programs (14) 


479 


143,475 


300 


92 


Technical 
Assistance 
Seminars and 
Workshops (5) 


192 


69,130 


360 


96 


TOTALS (29) 


1.027 


$474,726 


$462 





^Excludes participant travel for all seminars except Executive Component of the Court* 
** Because of the supervised internship, it is impossible to calculate the precise costs 
per participant day for the Phase 1 1 of the CEDP, 



ICM Activities Analysis, 1978 (cont) 



1 

; Activity (N) 


Number of 
Participants 


Total 
Costs* 


Cost Per 
Participant 


Cost Per 
Participant Day* 


Court Executive 
Development 










Phase 1 (7) 
Phase II 


292 
75 


$ 86,764 
149,472 


$ 297 
1,992 


$ 59 
NA** 


Juvenile Justice 
Management (3) 


73 


26,765 


367 


73 


Strengthening 
the Executive 
Component of 
the Court (1) 


20 


23,854 


1 193 


09/ 


Advanced and 
Continuing 

I Education 

j Programs (10) 


391 


84,023 


215 


66 


Technical 
Assistance 
Seminars and 
Workshops (5) 


171 


62,828 


367 


73 


TOTALS (26) 


1,022 


$433,706 


$ 424 





'Excludes participant travel for all seminars except Executive Component of the Court. 
♦•Because of the supervised internship, it is Impossible to calculate the precise costs per 
participant day for the Phase II of the CEDP. 
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the final budget. In the event that the Executive Director feels the need to modify or 
reallocate budget line items during the fiscal year, he refers the matter to the Board for 
a decision. According to I CM staff, the budget process is ''slightly out of kilter" in that 
the Institute's fiscal year coincides with the calendar year, whereas the LEAA funding, 
which currently constitutes 39% of the budget, runs from July to June. 

Organization: The Institute's executive staff consists of an executive director, an 
assistant executive director, and two department directors and a program coordinator. 
While the institute has by-laws and policies developed for travel and personnel, it does 
not have a comprehensive set of administrative policies. Similarly, there are procedural 
checklists for workshop development and logistics, but there is no comprehensive pro* 
cedural manual. The executive director delegates authority and responsibility among 
the executive staff. 

As we mentioned previously, the Institute's statement of mission and functions 
is currently under review. A de facto statement of its mission and functions is contained 
in the annual report and brochure. 

The management information system maintained by the Institute consists chiefly 
of budget and fiscal information, including staffing allocations, program performance 
information, and participant lists. While it is not an elaborate system. Institute officials 
are able to find what they need to know to manage the programs. 

Personnel: Employment policies have been developed specifically for the Insti- 
tute* They contain provisions for recruitment, salary classifications, salary increases, 
benefits, etc. There are no explicit provisions for discipline or grievance. Clerical and 
semi-professional staff evaluations are prescribed by the Employee Performance Re- 
view Policy and take place every six months. Professional staff evaluations take place 
once a year as part of the annual budget cycle with the Board of Trustees. 

Clerical staff and semi-professional staff are recruited and hired from the general 
population in the Denver metropolitan area. Salary classifications correspond to those 
of the city and county governments. Professional staff are recruited and hired on a 
national basis for a particular job designated by the Institute. Institute officials main- 
tain that IC^I has taken affirmative action steps to recruit and hire minorities— in- 
cluding the specification that more than 50% of ail candidates be minority, and that 
selection criteria be modified for minority applicants. At the present time, the Insti- 
tute has no minorities on the staff, but during the past two years two minorities 
have been hired who later resigned at employment intervals of three months and one 
year. 

Fiscal Management: Fiscal management of the Institute is under the direction of 
the Director for Administration and Planning, and is managed by an accountant and 
an administrative clerk. All budgets are approved by the Board of Trustees. Books are 
kept as in a for-profit organization (even though the Institute is nonprofit). A project 
cost accounting system is used, with subsidiary ledgers for each project by revenue 
and expense categories- Control points in the system are accounts receivable, expense 
control, income control, the checking account, and work in progress. An annual audit 
is performed by an outside firm. All checks are signed by two staff. Payroll is handled 
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by the accounting staff, with the requirement that all checks have two signatures. All 
Federal checks are deposited in a non-interest bearing checking account. All accounts 
are maintained by project, but restricted and unrestricted funds are comingled. The 
few travel advances are paid from a general ledger account. An imprest account of $50 
IS maintained for petty cash disbursements. 

Property management and inventory are maintained by the accountant; the inven- 
tory is performed on an annual basis. The Director for Administration and Planning 
handles all equipment rentals and purchases. Maintenance contracts are in force for the 
typewriters, copy machine, and postage meter. 

Locator Systems: The Institute has a moderate-sized library which was invento- 
ried and arranged by a librarian and is now maintained by the bookkeeper. Volumes 
and periodicals in the library are concerned largely with court systems, criminal justice 
systems, and other topics related to c ^urt management. While the materials appeared 
to be easily accessible and kept up to date, ICM staff admitted that they did not regu- 
larly use this resource. 



Challenges and Future Directions 

While the Institute is currently re-assessing its goals and objectives, and a clear 
picture of future directions is still a few months away, ICM officials were able to 
identify a number of needs which must be met; 

■ More research and understanding about the executive side of manage- 
ment is required, particularly in view of the possible trend to\Nard 
court unification. In this regard, a ""standards" approach is not ade- 
quate—the human element must also be considered. 

■ Further development and expansion of the executive team approach 
to courts training is necessary, i.e., bringing together the presiding 
fudge and the court manager in the same seminar sessions. 

■ For the 1980^s, court management training will tend to separate the 
court operations aspect from the executive management aspect. 

■ While a clearer picture of the universe of clerks of court is needed, 
it appears that training of this group is an important objective because 
in many cases they are the de facto managers of the courts. 

■ According to Institute officials, ICM has no way of knowing whether 
its graduates have the skills required. They are looking to the current 
evaluation of Courts Training Projects for some mdication of how to 
go about assessing this effect. 

In addition these needs. Institute staff also identified some potentially useful 
avenues of exploration, including a follow-up on its eariy graduates, courses for college/ 
university professors in court management, and identifying and officially recognizing 
outstanding court systems. 
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Training Objectives/ 

Perceived Effectiveness 

ICM and the instructors sun/eyed are in agreement as to the priority objectives of 
training, but the instructors demonstrate an unenthusiastic perception of what they are 
achieving {Tabic 8E). 

While both participants and instructors give moderately high ratings to the effec- 
tiveness of ICM, in terms of the multiplier effects listed in Table 9E, lower ratings show 
up in the following areas: 

■ Thfrty-four percent of the trainees do not share training materials with 
others in their jurisdiction; 

■ Forty-six percent of the participants do not attempt to make any per- 
sonal chants in the way they perform their duties as a result of train- 
ing;and 

■ Twenty-two percent of the trainees do not recommend ICM to others. 

While the relevance of ICM training gets generally high praise from participants 
and instructors surveyed, more individuals are critical of the training programs than at 
any of the other seven institutes. 

As for soundness of training design, participants and instructors alike give it "pass- 
ing grades'' (see Table lOE). 



Training Setting, 

Techniques. and Faculty 

In judging the adequacy of the training setting at ICM, participants and instruc- 
tors are generally complimentary except for one critical area. Some feel there is a poor 
mixing of trainees in terms of their professional roles and experience (see Table HE). 

The instructors are generally laudatory of training events management, but 46 per- 
cent express dissatisfaction with the degree of orientation and training they are given 
and, to a lesser extent, with a failure to receive appraisals of their performance. (See 
Table 12E). (These data agree with our initial finding during the Institute visit that 
ICM does not adequately assist instructors in setting performance objectives and 
instructional methodologies.) 

Instructors and participants agree that lectures and discussion groups constitute 
the most used teaching methods {Table 13E), and trainees find lectures far more use- 
ful than discussion groups. They do, however, attach value to informal discussions with 
peers and faculty outside the classroom setting. 

Faculty are given, high ratings by participants in the areas of substantive knowl- 
edge, practical experience, and teaching ability (Table 14E), and the instructors tend 
to be invited back quite often [Table 15E). 
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TABLE 8E 

INSTRUCTORS' PERCEPTIONS OF . EARNING OBJECTIVES (ICM) 



Objective 



1. Update and increase partici- 
pants' substantive knowledge. 

2« Improve participants pro- 
ficiency in their roles, 

3. Enable participants to intro- 
duce new techniques and pro- 
cedures in their systems. 

4. Enhance participants' under- 
standing of their roles. 

5. Enable participants to in- 
fluence/promote change 
among other courts person- 
nel. 



6. Increase communication and 
consultation among profes- 
sional peers/colleagues. 



Perceived Achievement 

Average Category of 

Priority Rating Median 
(1 -High; 7- Low) Response 



2.48 



Fairly Well 



2,92 



Fairly Well 



3,16 



Fairly Well/ 
Very well 



3,28 



Fairly Well 



3.60 



Fairly Well 



4.48 



Fairly Well 



TABLE 9E 
MULTIPLIER EFFECTS (ICM) 



Aspect 



Type of Numb6r/% 
Respondent Responding 



Frequency of Response 



JNL Jk YES NO No response 
N % N % N 



Wish to return to 
institute. 

Perception of par- 
ticipants' desire to 
return to institute. 

Perception of par- 
ticipants' actual 
return to institute. 

Share institute 
materials. 

Perception of par- 
ticipants' use of 
institute materials. 

Use institute materials. 

Attempt to make insti- 
tute*spurred per- 
sonal changes. 

Attempt to make insti- 
tute*spurred organi- 
zational changes. 

Recommend institute 
to others. 



Participant 60 98 52 87 8 13 1 

I nstructor 28 1 00 26 93 2 7 — 

Instructor 25 89 25 100 - - 3 

Participant 59 97 39 66 20 34 2 

Instructor 25 89 24 96 1 4 3 

Instructor 27 96 22 81 5 19 1 

Participant 56 92 30 54 26 46 5 

Participant 56 92 34 61 22 39 5 

Participant 59 97 46 78 13 22 2 
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TABLE 10E 
SOUNDNESS OF TRAINING DESIGN (ICM) 



Category Frequency of Response 

Type of Number of Median 

Training Design Component Respondent Responding Response Not at all Somewhat Fairly well Very well Thoroughly No Response 



1. 


Degree based on 


Participant 


58 


Very well 


3 


7 


14 


19 


15 


3 




profession's needs. 




(95%) 




(5%) 


(12%) 


(24%) 


(33%) 


(26%) 








Inctriirtnr 


97 


Won/ wall 

very Well 




1 
1 






9 


1 
1 














14%) 






I7%1 

\I70I 




2. 


Degree indii'idual's 


Participant 


60 


Fairly well 


10 


13 


19 


16 


2 


1 




needs are considered. 




(98%) 




(17%) 


(21%) 


(32%) 


(27%) 


(3%) 








Inctriirtnr 

IlldtlUUlUl 


Oft 


very wen 






7 

1 


17 

1/ 


0 










llUUyo) 










IblTDl 






3. 


Degree expected out- 


Participant 


59 


Fairly well 


10 


10 


15 


16 


8 


2 




comes presented at 




(97%) 




(17%) 


(17%) 


(25%) 


(27%) 


(14%) 




T 


outset ot course. 


insirucior 


Oft 


very weii 


1 

1 


0 

0 




c 
0 


c 

t> 


2 


si 

if 






\\)61b] 












liy%) 




i 

% 


uegree learning od- 


rariicipani 


C7 
0/ 


rainy well 




15 


in 


16 




A 




jectives clear and 




(93%) 




(12%) 


(26%) 


(18%) 


(28%) 


(16%) 






succinct. 


Instructor 


26 


Very well 




4 


7 


14 


1 


2 








(93%) 






(15%) 


(27%) 


(54%) 


(4%) 




5. 


Degree objectives are 


Participant 


58 


Fairly well 


13 


13 


11 


15 


6 


3 




demonstrable. 




(95%) 




(22%) 


(22%) 


(19%) 


(26%) 


(10%) 








Instructor 


26 


Fairly well 


1 


3 


12 


9 


1 


2 








(93%| 




(4%) 


(12%) 


(46%) 


(35%) 


(4%) 




6. 


Degree training 


Participant 


59 


Fairly well 


12 


12 


10 


15 


10 


2 




provides opportunities 




(97%) 




(20%) 


(20%) 


(17%) 


(25%) 


(17%) 






to practice what is 


Instructor 


28 


Fairly well 




4 


11 


11 


2 






taught. 




(100%) 






(14%) 


(39%) 


(39%| 


(7%) 





TABLE 10E (Continued) 
SOUNDNESS OF TRAINING DESIGN (ICM) 



Category 

Type of Number of Median 



Frequency of Response 



Training Dssign Component Respondent Responding Response Not at all Somewhat Fairly well Very well Thoroushly No Response 



informed of progress. 



materials, and activities 
cohesive. 



evaluation useful. 



g to provide input to 
faculty. 



graded and modified 
in relation to current 
state of the art. 



Participant 


54 


Not at all 


26 


12 


7 


7 


2 


7 




(Ml 




(Wo) 




(i3%l 


(13%) 


(4%) 


Instructor 




rairiy wen 


I 


7 

1 


A 

9 


4 


2 


4 




(OD7o) 




(o/ol 




(38%) 


(17%) 


(8%l 


Participant 


59 


Very well 


1 


9 


8 


27 


14 


2 


Instructor 


15/ All 






(15%) 


14 in/ 1 

(14%) 


(46%) 


(24%) 


26 


Very well 




2 


6 


13 


14 


2 










(7%) 


(23%) 


(50%) 


(19%) 


Partipirtant 
rai ULi|Jaill 


0/ 


Cn!.!.. ,.,.11 

rainy well 


1 A 

14 


8 


10 


20 


5 


4 


Instructor 


(93%) 




Uja)) 


(14%! 


(lo%| 


(35%) 


(9%) 


25 


Very well 




6 


3 


12 


4 


3 




(89%) 






(24%) 


(12%) 


(48%) 


(16%) 


Participant 


60 


Very well 


1 


10 


8 


26 


15 


1 




(98%) 




(2%) 


(17%) 


(13%) 


(43%) 


(25%) 


Instructor 


26 


Very well 




1 


6 


11 


8 


2 




(93%) 






(4%) 


(23%) 


(42%) 


(31%) 



TABLE 11E 

ADEQUACY OF THE TRAINING SETTING (ICM) 



Type Of Number/% 
Aspect Respondent Responding 

1. Similarity of participants' Participant 59 
roles/experience (97%) 

Instructor 28 

(100%) 

2. Sufficiently small classes Participant 59 

(97%) 

Instructor 28 

(100%) 

3. Sufficiently long courses Participant 57 

(93%) 

Instructor 28 

(100%) 

4. Match between partici- Participant 56 
pant needs and instruc- (92%) 
tor expertise Instructor 27 

(96%) 

5. Availability of instruc- Participant 58 
tors outside of class (95%) 

Instructor 27 

(96%) 

6. Sufficient number of Participant 60 
contact hours between (98%) 
participants and in- Instructor 27 
structors (96%) 

7. Reinforcement of be- Instructor 27 
havior/skiits (96%) 

8. Adequate support ser- Participant 60 
vices (98%) 

Instructor 27 

(96%) 

9. Adequate physical ac- Participant 61 
commodations ( 1 00%) 

Instructor 27 

(96%) 



Frequency of Respo nse 



Yes No Response 



*3Q 
OO 


26 


2 


(56%) 


(44%) 




21 


7 




(75%) 


(25%) 




51 


8 


2 


(86%) 


(14%) 




26 


2 




(93%) 


(7%) 




47 


10 


4 


(82%) 


(18%) 




23 


5 




(82%) 


(18%) 




43 


13 


5 


(77%) 


(23%) 




26 


1 


1 


(96%) 


(4%) 




54 


4 


3 


(93%) 


(7%) 




25 


2 


1 


(93%) 


(7%) 




55 


5 


1 


(92%) 


(8%) 




23 


4 


1 


(85%) 


(15%) 




24 


3 


1 


(89%) 


(11%) 




56 


4 




(93%) 


(7%) 




27 




1 


(100%) 






59 


2 




(97%) 


(3%) 




27 




1 


(100%) 
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TABLE 12E 

MANAGEMENT OF TRAINING EVENTS (\CM) 



CTP Instructor Response 



Aspect 



1 . Clarity of lines of authority/ 
responsibility 

2. Adequacy of administrative 
procedures 

3. Instructors appraised of per- 
formance/expectations 

4. Instructors given orienta- 
tion/staff training 

5. Program modification based 
on assessment 



Number 
Responding 



_N 
27 

28 

28 

26 

22 



96% 
100% 
100% 
93% 
79% 



Yes 



ii 

25 
28 
23 
14 
21 



% 
93% 

100% 

82% 

54% 

95% 



No 



N 



5 
12 
1 



% 



7% 



18% 
46% 
5% 



ERIC 
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TABLE 13E 
FREQUENCY OF METHODS USED (ICM) 



■Method . Madtan Response 

CTP Participant CTP Instructor 



1. 


Lectures 


Most of the time 


Often 


2. 


Discussion groups 


Often 


Often 


3. 


Panel discussions 


Not at all 


Some 


4. 


Case studies 


N/A 


Some 


5. 


Role playing, simulation 


Not at ail 


Some 


6. 


Videotaping 


N/A 


Not at ail 


7. 


Moving pictures 


N/A 


Not at all 


8. 


On-the-job training 


N/A 


Not at all 


9. 


Computer Assisted 


N/A 


Not at all 



Instruction 
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TABLE 14E 
PERCEIVED FACULTY COMPETENCE (ICM) 



Aspect 



Number 



Category Totally Minimally 
of Median Unknow- Knowledgs- 
Response ledgeable able 



Quite No 
Adequate Knowledgeable Expert Response 



Degree of their sub- 58 Quite 
stantive Itnowledge. (95%) Knowledgeable 



13%) 



6 

(10%) 



27 

(47%) 



23 

(40%) 



Extensivenessof 58 Quite 

their practical ex- (95%) Knowledgeable 

perience. 



4 

(7%) 



6 

(10%) 



25 

(43%) 



23 

(40%) 



Teaching ability. 



Number times 
called upon staff/ 
instructors since 
training for followup 
technical assistance. 



56 

(92%) 



60 

(98%) 



Quite 1 2 

Knowledgeable (2%) 14%) 

Not at Once or 

all twice 



Not at all 



42 

(70%) 



13 28 
(23%) (50%) 



12 5 

(21%) 

No 



Periodically Often Continually Response 



(15%) 



6 

(10%) 



3 

(5%) 



TABLE 15E 

FACULTY FOLLOWUP/CONT. RELATIONSHIP (ICM) 



Called upon by CTP 
Number of Timoi Participants Invited back to Institute 

N % N % 

None 4 15 5 19 

I -5 11 42 12 44 
6-10 2 8 8 30 

II -25 3 11 2 7 
Greater than 25 6 24 0 0 



INTEREST IN TEACHING AGAIN 
YES (%) NO {%) 
100 0 
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6. NATIONAL COLLEGE Of DISTRICT ATTORNEYS 



The National College of District Attorneys located at the Bates College of Law of 
the University of Houston, was visited on December 19 and 20, 1978 by a two-person 
Interview team. The site visit was initiated with the Administrative Assistant, who 
described the management procedures of the office and introduced the team to the 
staff. 

The team met with the Dean and Assistant Dean to explain the purpose of the 
study and the goals of the visit. A detailed interview reviewing the history of the insti- 
tution, its present goals and staff was conducted with the Dean, using the project direc- 
tor questionnaire as a guide. Subsequent inten/iews were held with the Assistant Dean, 
Director of Training, Director of Course Development, and Research Attorney. Inter- 
views were also conducted with the Financial Manager, Communications Manager and 
Registrar. At the conclusion of the visit, the project team requested additional informa- 
tion which was not readily available at that time. — 

A number of documents were reviewed, including faculty handbooks, procedures 
manual for personnel. Curriculum Committee minutes, as well as National College 
catalogs and brochures. Subsequent to the visit, staff members completed instruments 
regarding staff, faculty and financial statistics to satisfy the information needs. 



History of 

the Institution 

The National College of District Attorneys was created in response to recommen- 
dations made by the President's Commission on Law Enforcement and Administration 
of Justice. Several groups, including representatives from the National District At- 
tomeys Association, the American Bar Association, the American College of Trial 
Lawyers and the Intemational Academy of Trial Lawyers, met in 1969 to consider the 
creation of a school for prosecutors. As a result, the first program, the Career Prose- 
cutor Course, was offered in the summer of 1970. 

Several foundations contributed financial assistance. Two of the regents, Louis B. 
Nicholas and Leon Jaworski, were instruments! in the initial fund-raising efforts. 

In 1969, the College accepted an offer from the Dean of the Law School at the 
University of Houston to use office space and facility support at the Law School. The 
College has been housed there since that time. 
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In 1970, the first Dean, George Van Hoomissen, was appointed by the Board of 
Regents to provide full-time leadership in developing the academic program of the 
institution. The present Assistant Dean, Robert S. Fertitta, served as Acting Dean 
during 1973, and in 1974 John Jay Douglass, the present Dean, was appointed. 

The Board of Regents, the governing body of the College, meets annually to 
review the progress of the College and staff performance, and to provide guidance to 
the Dean in matters of budget, curriculum, format and priorities. This meeting is 
the only direct and regular contact between the entire Board and the Dean, but the 
Dean discusses issues and seeks advice with individual members of the Executive 
Board by phone. 

The Board consists of 11 members including four repre^ntatives of the National 
District Attorneys Association, two from the American Bar Association, two from the 
American College of Trial Lawyers, one from the International Academy of Trial 
Lawyers, one from the University of Houston, and one regent emeritus. 

The present staff consists of a Dean, an Assistant Dean and Director of Finance, 
Director of Course Development and Research, Director of Training, Assistant 
Director of Training, Research Attorney, Communications Manager, Financial 
Manager, Administrative Assistant, Assistant Communication Manager, a Registrar, 
four secretaries, a bookkeeper, a clerk-typist, and several student interns. 

Since 1971, LEAA has provided a series of grants to enable the College to 
conduct the Career Prosecutor Course and other short courses which the College de- 
veloped in October 1971- In 1973, the Executive Prosecutor Course was added. 

Early in its history, the College instituted short courses held at various locations 
around the country. By 1977, the College had conducted 22 short courses which had 
trained 1,500 prosecutors. Under a special provision of the Crime Control Act, LEAA 
funds were provided for seminars on the prosecution of organized crime. 



Goals and Objectives 

of the Institution 

The mission of the College is to improve the skills of the prosecutor. The primary 
goals inherent in this mission are to assist in the definition of the prosecutor as a vital 
and unique member of the legal profession and of the criminal justice system; to pro- 
mote the identification and understanding of the roles and responsibilities of the office 
of public prosecutor; to aid in the improvement of the legal, technical, and ethical art 
and science of prosecution; and to establish contact with as many prosecutors as pos- 
sible» In addition, the College encourages a professional attitude for support personnel 
in the prosecutor's office. 

The goals are stated in the presentation paper submitted for fund solicitation and 
in the academic catalog which is published annually to describe courses offered during 
that year. 
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Programs 



Trends: The number of LEAA-funded programs and participants has remained 
fairly consistent over the past four years. In 1977 the level of participation of LEAA 
programs was relatively lower, but the number of participants in programs supported 
by other funds was higher, due to a large number of programs funded by other federal 
and private grants. 



PROGRAM STATISTICS 





1975 


1976 


1977 


1978 


Number of LEAA Programs 


13 


14 


14 


12 


Number of Participants 


942 


1,103 


824 


1.084 


Number of Otiier Programs* 


3 


5 


13 


5 


Number of Participants 


207 


289 


442 


268 



*Other programs include courses held from finar)cial sources other than the LEAA 
Courts Training Grant, i.e., HEW funds, special LEAA grants, and privets funds. 



More programs are focusing on specialty topics. The College recognizes that its 
audience is growing more sophisticated and attempts to adapt its programs to their 
expectations in terms of both content and format. The College believes tf.at in the 
future the students will need more training in evidence and in courses which focus 
on attitudes and philosophy of prosecutors. 

Current Programs: The present courses range in length from two and one-half 
days to three weeks. The subject matter includes one-subject specialized short courses 
as well as multi-faceted courses designed to present an overview and in-depth treatment 
of a variety of subjects. The courses are classified as either resident courses (held at the 
University of Houston campus) or short courses {held off campus in various locations 
around the country). 

The College holds three resident courses annually: 

Career Prosecutor Course: A three-week course which is an intensive study of the 
office of the prosecuting attorney. The course presents a chronological study of the 
role of the prosecutor from the criminal act to the return of the accused to society. 
(150 attendants) 

Executive Prosecutor Course: A one-week course designed for prosecutors or 
assistants serving in a policy-making decision capacity. (60 attendants) 

Prosecutor's Office Administrator: A three-week course taught in independent 
one-week segments which provides specialized management training for prosecutor's 
office administrators and managers. (25 attendants at each session) 
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In addition, two Trial Techniques courses were held at two different locations for 
four and one-half days. A combination of lecture and videotape demonstrations 
served as the key teaching method. (150 attendants at each session) 

Five specialized courses are scheduled for the 1978-79 academic year. Each course 
is conducted for three and one-half days. The topics are: 

» Prosecuting Crimes Against Persons; 

» Management in the Prosecutor's Office; 

■ Prosecuting Drug Cases; 

■ Prosecuting Crimes Against Property; and 

■ Law and Evidence for the Prosecution Advocate. 
The College trains approximately 1,500 persons annually. 

A Desk Assessment Course was designed to meet the needs of small offices and 
part-time prosecutors who are unable to attend the three-week Career Prosecutor 
Course. The first program. Constitutional Law, was initiated in 1977 and presently has 
227 enrol lees. The second, developed in 1978, The Roles and Functions of the 
Prosecutor, has 134 students, to date. Both courses were designed by the Research 
Attorney on the College staff. A third Desk Assessment Course on management, 
developed by the Director of Course Development and Research, will be ready for im- 
plementation in early 1979. 

Three seminars on organized crime are also being held during the 1978-79 
academic year. These programs have been funded since 1972 by a special LEAA grant 
under the Crime Control Act. 

The College also prepares a program with the Federal Bureau of Investigation for 
investigators employed in the prosecutor's office who have less than five years of 
experience. 

In addition to the programs mentioned above, the National College offers 
technical assistance to states in planning and developing training programs. The College 
staff may provide on-site assistance in planning or support assistance coordinated from 
the office in Houston. In this service, the College usually works with Prosecutor 
Training Coordinators. 

(3) Needs Analysis: While there is no systematic needs assessment performed at 
the National College, several steps are taken to help determine the training needs of 
the target audience. The Research Attorney engages in research which assists in identi- 
fying the educational needs of prosecutors, in developing new courses, and in modify- 
ing old ones. Several new courses, including those on narcotics, crimes against persons, 
and child abuse, were initiated as a result of this research. 

The College also depends on the Curriculum Committee as a resource for needs 
analysis, as well as the results of cdurse and faculty evaluations submitted by the 
participants. 
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The personal contacts which the College maintains with former students, faculty 
and active prosecutors also contribute to the needs analysis efforts. 

Curriculum Development: The two Deans are responsible for the overall co- 
ordination of curriculum development. The Curriculum Committee initiates the course 
selection process at the annual meeting, and its decisions are reviewed and refined 
by the College staff and submitted to the Board of Regents for approval. 

The Curriculum Committee consists of fifteen members who are faculty, faculty 
advisors, former students, representatives of sponsors of the College and College staff. 
The Committee meetings are scheduled annually in the fall after the summer sessions 
and before the Board of Regents meetings. As part of its review, the members are 
given a source book, which includes the evaluation results of the summer courses. The 
Committee meets in a plenary session and later breaks into three groups to discuss the 
major course areas covered by the College: the Career Prosecutor Course, the Executive 
Prosecutor Course, and short courses. 

The Committee sets up tentative programs and the two directors determine loca- 
tions for the courses.The courses are then assigned to one of the staff attorneys. The 
assignments are made approximately eighteen months in advance of the course date 
so that adequate time is allowed for preparation regarding course materials, faculty 
selection, budget and other related tasks. 

The tentative program is prepared by the attorney 90 days before the course. The 
program is then submitted to the Dean for his approval. He sends his reactions to the 
Associate Dean within 48 hours, and feedback is then provided to the staff attorney 
in charge of preparation. 

While all the staff members share in the responsibility for resident courses, the 
Director of Training is responsible for the courses on Trial Techniques and those on 
organized crime. The Director for Course Development and Research supervises the 
courses in management and fraud. The Re^arch Attorney oversees some of the short 
courses, as well as the Desk Assessment Course. 

Faculty Selection and Development: The National College cities several criteria 
for faculty selection, including expertise in a field, good speaking skills, ability to limit 
and adhere to a specific topic, and willingness to use the methodologies recommended 
by the College. 

The College emphasizes the fact that speakers are not chosen solely for their 
stature, but for knowledge in their field and the ability to communicate effectively. 

A faculty file is maintained which is arranged according to conference topic. A 
dossier is built on each prospective faculty member before an invitation to speak is 
extended. The staff may contact colleagues in the prospective faculty member's field 
for recommendations, or review videotapes of a lecture presented by that person. 

Faculty are also chosen in terms of geographical locations with priority given to 
those located near a program site to avoid large transportation expenditures. 
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The College does not use the same faculty repeatedly in order to maintain a 
diversified roster and to avoid dependence on a small number. However it does 
maintain a core faculty who are most frequently recruited to teach. 

Each faculty member receives a handbook for each course well in advance of the 
conference. This handbook contains course objectives, topic listings, and training 
materials as well as information regarding hotel and travel expenses. A faculty member 
IS required to submit a paper on the course topic to the College 60 days in advance of 
the program. 

In most cases, the faculty find it difficult to meet before the program, since they 
teach on different days. In a module course, the team members are encouraged to 
make conference calls to get a consensus regarding topics and speciffc duties of each 
member. In the management course, the half^iay free time given to students is used as 
faculty meeting time. 

Faculty advisors, who are graduates of the College, are used to lead the small 
group workshop sessions of the Career Prosecutor Course. Outside consultants are 
often used for preparation of scenarios which are components of some of the courses. 

The National College provided the evaluation team with information on 21 
faculty members who are considered to be the core faculty. As indicated earlier, 
faculty is rotated frequently and files are maintained on hundreds of potential 
teachers. Of the 21 provided as the core, 20 are white males, and one is a black male. 
Four faculty members hold LLM. degrees, fifteen hold either LL.B. or J.D. degrees, 
one holds a M.A. in Criminal Justice, and one is a professor of pathology. 

The College did not provide figures on the tenure of faculty. However, of 9 
faculty with teaching experience, the average was 12 years of teaching. Of the 19 
members with experience in the field of justice, the average is 10 years of experience. 

Permanent Staff: The Dean of the College is appointed by the Regents and is 
the principal officer responsible for the execution of policy. The Dean supervises 
overall program development, staff selection and personnel policy and procedures. 

The Associate Dean assists the Dean in all activities, acts for him in his absence, 
and is responsible for the financial and administrative operations of the College! 

The Director of Training has the primary responsibility for developing regional 
and specialized short courses. He also supervises curriculum development, educational 
materials, and evaluation measures for those programs. In addition, he assists in the 
presentation of the resident courses. The Assistant Director of Training aids him in 
these duties. 



The Director of Course Development and Research coordinates publication 
projects, and assists in the preparation of educational training materials for courses. 
The Research Attorney, who reports to the Director of Course Development, is re- 
sponsible for research activity for current and potential programs, for the development 
of the Desk Assessment Courses, and the review of all publications. 
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AM of the law-trained staff assist in preparing and evaluating the National College 
programs. None of them is pemnitted to engage in private law practice while employed 
by the College, although they may teach, consult, or lecture. 

The Communications Manager is responsible for the technical preparation of all 
course materials and publications of the National College, including program outlines, 
handbooks, brochures and newsletters. In addition, the Manager maintains the College 
mailing list. The Assistant Communications Manager prepares the semi-annual 
Bibliographic Guide for Prosecuting Attorneys, maintains the college library and 
resource center, and coordinates audio-visual services for the College. 

Other members of the administrative staff include the financial manager, who 
maintains the financial books and budget; the administrative assistant, who manages 
the non-education functions of the College; and the Registrar, who processes the 
applications and student records. 

The administrative staff is recruited through newspaper advertisements and em- 
ployment agencies. The law-trained staff are often recruited directly from law schools 
through placement offices, or by recommendations of a state training director. 

Regarding the characteristics of the staff members, at the time of our visitation, 
the professional staff consisted of seven men and four women, all of whom were white. 
The Dean holds a LLM. degree; five staff members hold J.D. degrees; three hold either 
a B.A. or B.S. degree; and one holds an A.A. degree. The average tenure of the profes- 
sional staff is three years. The average age is 36 years. Both the Dean and Associate 
Dean have had a number of years of teaching experience, as well as extensive experi- 
ence in the field of justice. 

The College also employs one student administrative assistant (white, male) who 
holds a J.D. degree. 

The support staff was composed of four full-time and two part-time females. Of 
the six, three were black, two white, and one oriental. One of the secretarial staff has 
a B.A. degree and the remainder are non-degreed. The average age is 31 years and the 
average tenure is 7 months. The College attributes the high tumover rate among 
support staff to low salaries. The average salary is $7,654, which is said to be low for 
support personnel in the Houston area. 



Effectiveness 

Marketing of Programs and Selection of Trainees: The Communications Man- 
ager supervises the marketing efforts of the College through the publications and 
mailing list. The mailing list consists of 5,800 lawyers and investigators. Many of the 
listings are sent to prosecutor's offices rather than to specific individuals. The bro- 
chures are sent to all District Attorney's offices and to 284 top law offices to the atten- 
tion of the Investigation Section. There is also a direct mailing to all new District At- 
torneys on a name basis. 
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M • ^"^f "^^'''"S list IS maintained by an outside computer service with which the 
National College contracts. Items on the list can be retrieved by region and category 
of profession. The College sends two mailings for each seminar, since they have found 
the first mailing often gets lost within the office. 

The admissions standards for the course often depend on the content. For example 
the Career Prosecutor Course is limited to those who have two or mor« years of experi- 
ence in a prosecutor's office and who have previously attended a local, state, or 
?h?In"f« P'-osecutors. Preference for the Executive Prosecutor Course is 

ProicutcJ? Cou'!^ ° ^ policy-making position and who are graduates of the Career 

Among the criteria for selection of participants, the College considers geographic 
distribution and gives priority to those students who are receiving financial aid from a 
state planning agency or other state or local organization. 

Monitoring and Assessment of Programs, Faculty, and Trainees: The National 
College evaluates its programs through faculty and student critiques administered at 
every course. A more extensive evaluation of faculty is used during the summer pro- 
grams. There are also some pre- and post-tests being administered this year. The staff 
also monitors the courses carefully, being particularly observant of new faculty. 
Follow-up critique fomrjs are sent to students who complete the Career Prosecutor 
Course and to their supervisors. The results of all of these evaluations are used for 
future program planning. 

However, there appear to be some serious weaknesses in the College's monitoring 
and assessment practices. Among the NCDA training participants interviewed at the 12 
randomly selected case study sites, a significant number complained about sexist 
attitudes and comments of College staff and faculty and their insensitivity to members 
of minority groups. 

Another area which seems to need attention, especially at NCDA's regional work- 
shops, is attendance and awarding of Continuing Legal Education credits. A number 
of participants commented critically on NCDA's failure to require and enforce atten- 
dance at its training programs or to withhold CLE credits from the absentees. Our on- 
site observation of one NCDA training program confirmed the existence of these prac- 
tices. 

In 1974 the College commissioned an evaluation, which was to focus on the activ- 
ities for the 1974-75 grant period. The report, by Joseph L. Carr, B.J. George, Jr., and 
Donald Webber, focused on the College's organization, programs, services, and faculty. 
The recommendations included the improvement of faculty development, curriculum 
planning, and course format. The College responded to these suggestions, which are 
reflected in the present organization and programs. The Dean, at this time, however, is 
more interested in an evaluation which deals with student retention and program 
impact and which offers suggestions regarding needs assessment and marketing 
strategies. 
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Oiher InstUurtonal 
Activities 

Research: The National College employs a Research Attorney primarily to keep 
programs current and to provide data regarding needs for new courses. The College 
maintains a Resource Center, a clearinghouse for educational materials from prosecu- 
ting attorneys' offices and prosecutor training coordinators from ail over the country. 

The College would like to expand its research capabilities, hat lacks the necessary 
funds. 

Publications: Several texts have been published by the National College: 

■ The Right of the People to be Secure-An Examinaiion of the Fourth 
Amendment, 2d. ed. by Judge Charles E. Moylan, 1977 

o A Compendium on Child Abuse and Neglect, 1977 

■ Incompetency Issues in Criminal Trials, 1977 

• The Constitutional Law DeskboolQ, 1978 

■ Constitutional Law in Texas, 1978 

a A Bibliographic Guide for Prosecutors, 1977 

The Bibliographic Guide has been distributed to over 2,500 prosecutors at no 
charge. 

The six texts that are used in the Role of the Prosecutor Desk Assesvnent Course 
are also available separately for purchase. 

• The Prosecutor in America 

■ Discretionary Authority by the Prosecutor 

• Ethical Consideration in Prosecution 

» Prosecutorial Relationships in CriminalJustice 

■ Pretrial Problems of the Prosecutor 

■ Special Problems of Prosecutors 

The College publishes Update, a quarterly newsletter, as well as a Faculty 
Newsletter, and a Master Calendar of Programs for Prosecutors. 



EKLC 



11-^2 



22 



o 



External Relations: The National College shares some of its facilities and servfces 
With the National College of Criminal Defense Lawyens and Public Defenders 
(NCCDLPD) which is located in the same building. The colleges do not collaborate on 
programs because they claim that it is too difficult to do this within an adversary 
system. The two colleges also state that their schedules, speakers, and types of pro- 
grams are not conducive to joint programs. 

The College is aware of the National Institute for Trial Advocacv's (NITAI 

ffi'u J"st4°S "°u ^'^^ ' -'^^ regard to^L5.ing'trfa 

S=p'P<;2:;rn:^^r^^^^^^^ ^^^'"'^ them is limited. NITA also^ims 

Regentt!^""^' «"hanced by its four principal sponsors on the Board of 

L PAl'^Tho^n".?^' College recognizes, in its publications, the financial assistance from 
LEAA. The College feels that LEAA, especially the Adjudication Division Sould iS 

ToL nf nr^ ^1^^^:°^'^,^ ^"^^ ^ S^^tir control and pmd'en^ in ^ir 

cho.c» of grantees. The College feels there is excessive duplication and a lack of Com- 
munication among all the training programs. « «ia a lacK or com 

Alumni Development: Alumni receive all brochures which are published as well 
as Update the quarterly newsletter. Every other year, they receive an Alumni Direc- 
tory which IS covered by the $5.00 membership fee. If an alumnus has not attended a 
course in the last 18 months, however, his/her name is deleted from the mailing list. 

An Alumni Association, which consists of graduates of the Career Prosecutor 
Course and the Executive Prosecutor Course, numbers approximately 1 200. The 
Association has established an alumni scholarship fund. 



Ptogram and 

Related Costs 

Budget History: Over the past three years. LEAA has decreased funding to the 
College. At the same time, sponsor funding has also been slowly decreasing. Some 
recovery was made through increased tuition revenue. The College has $450,000 in 
time deposits and capital savings. 



Revenue 


1976-77 


1977-78 


1978-79* 


Tuition/Fees 
LEAA Grant 
Sponsor Funding 

Total 


$ 87,350 
500,000 
55,555 


$207,736 
450,000 
50,000 


1 
1 

$129,200 
350,000 
38,978 


$642,905 


$707,736 


$518,178 



'Current Fiscai Year's revortues are astimaas. 
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Activities Analysis: The Career Prosecutor Course, a three-week resident pro- 
gram, was attended by 150 students, at a cost of $327.25 per student. This was the 
highest cost of any course offered by the College. The lowest cost per student for a 
resident course was for the Prosecutor's Office Administrator Course III. A total of 
36 students participated at a cost of $84.33 per student. 

The costs for specialized short courses are well represented by the average 
amounts {see chart below). The highest cost per student was $77.54 for the Manage- 
ment in Prosecutor's Office course which was attended by 67 students. The lowest cost 
per student $40.35, was for Prosecuting Crimes Against Persons, which was attended 
by 141 students. 

The two courses offered on Trial Techniques were very similar in both attendance 
and cost per student. Therefore, the averages presented below are representative of 
both courses. 



CONFERENCE STATISTICS 



1 

Category 


Total No. 
Conferences* 


Avg. Cost 

per 
Conference 


Avg. No. 
Students 


Avg. Cost 

per 
Student 


1 Avg. Cost 
per 
Student 
per Day 


Resident Courses 


4 


$18^58 


71 


$257 


$33 


Short Courses 


5 


$ 5,232 


94 


$ 56 


$22 


Trial Techniques 


2 


$ 6,150 


123 


$ 50 


$13 



indicates conferences with available data. 



Management 

Policy Planning: The Board of Regents is primarily responsible for the basic 
policy of the College which the Dean executes. In addition to the annual Board 
meeting, the Dean maintains individual contact with members throughout the year 
when a question of policy or operations arises. 

The responsibilities of each of the directors, the Associate Dean, and those 
working with them are clearly delineated and the lines of authority well structured. 

Budgeting: The Dean and Associate Dean determine the budget each year and 
the financial manager and bookkeeper administrate the records. The Dean is involved 
in fund raising in order to seek additional sources of income for the College. 
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The policies and procedures of the College are well documented and reflect a 
well-organized institution in terms of management, programs, and personnel. One of 
the reasons that the College has good organization is the stability of the top admin- 
istrators. The Dean has been with the College for 5 years, the Assistant Dean has 
7 years tenure, the Director of Course Development and Research has 5 years tenure, 
and the Research Attorney has worked for the College years. This stability al- 
lows continuity of policy, management and program development. As a result, the 
College has policies and procedures manuals for both staff and faculty which cover in 
detail the appropriate areas. The handbook which is distributed to faculty before con- 
ferences prepares the instructor for the course and introduces him to the procedures 
of the College. 

The sourcebook on summer courses which is compiled and distributed at the Cur- 
riculum Committee meeting is an appropriate springboard for curriculum planning. 

Information on past programs in temns of content, participants, evaluation, and 
costs is readily accessible. 



Challenges and 

Future Directions 

The National College would like to expand and improve the quality of its 
programs and sen/ices through additional faculty training, improving audio-visual 
techniques, increasing the number of publications, better targeting of students and 
improving outreach strategies. The College hopes to reduce costs through improved 
planning for programs and through requests to airlines and other funding sources for 
scholarship money for student transportation. 



Training Objectives/ 

Perceived Effectiveness 

The teaching objectives of surveyed NCDA instructors coincide with the stated 
objectives of the college, and the instructors believe the success rate is high in achieving 
those objectives (Table 8F). This perception tends to be borne out by participant and 
'Tl?^''^"/ ^'^"^ °' relevance of the training and their ratings of the effectiveness 
of NCDA training as reflected by the multiplier effects shown in Table 9F. The only 
soft spot IS the 61 percent of NCDA participants who do not attempt to make organi- 
zational changes as a result of training experiences, but that probably more accurately 
reriects the status of the trainee in his/her organization than the effectiveness of the 
training. 

However, there are mixed reactions by participants and instructors to questions 
concerning soundness of training design. Participants are somewhat critical of NCDA's 
failure to consider the needs of the individual trainee, to keep the individual informed 
of his/her progress, and to provide useful feedback, and opportunities for evaluative 
comments from participants to faculty. In a number of other aspects of training de- 
sign, only passing grades" are accorded by participants and instructors. (See Table 
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TABLE 8F 

INSTRUCTORS' PERCEPTIONS OF LEARNING OBJECTIVES (NCDA) 



Objective 



1 . Improve participants' pro- 
ficiency in their roles. 

2. Update and increase partici- 
pants' substantive knowledge. 



3. 



4. 



Enable participants to intro- 
duce new techniques and pro- 
cedures in their systems. 

Increase communication and 
consultation among profes- 
sional peers/colleagues. 



5. Enhance participants' under- 
standing of their roles. 

6. Enable participants to in- 
-fluence/promotechange 
among other courts person- 
nel. 



Perceived Achievement 

Average Category of 

Priority Rating Median 
(1-High;7>Low) Response 



2.38 



Very well 



2.81 



Very well 



3.15 



Fairly Well 



4.27 



Very well 



4.28 



Very well 



5.00 



Fairly Well 
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TABLE 9F 
MULTIPLIER EFFECTS (NCDA) 



Aspect 



Type of 
Respondent 



Number/% 
Responding 

N % 



Frequency of Response 

-J^ES_ NO No response 
N % N % N 



Wish to return to 
institute. 

Perception of par- 
ticipants' desire to 
return to institute. 

Perception of par- 
ticipants' actual 
return to institute. 

Share institute 
materials. 

Perception of par- 
ticipants' use of 
institute materials. 

Use institute materials. 

Attempt to make insti- 
tute-spurred per- 
sonal changes. 

Attempt to make insti- 
tute*spurred organi- 
zational changes. 

Recommend institute 
to others. 



Participant 108 100 94 87 14 13 



Instructor 

Instructor 
Participant 

Instructor 
Instructor 



39 89 38 97 1 3 5 

36 82 33 92 3 8 8 

108 100 94 87 14 13 - 

39 89 38 97 1 3 5 

41 93 34 83 7 17 3 



Participant 99 81 76 26 24 



Participant 99 39 65 61 



Participant 108 100 93 86 15 14 
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TABLE 10F 
SOUNDNESS OF TRAINING DESIGN (NCDA) 



Cdtegory 

Type of Number of Median Frequency of Reti)on$e 

Training Desijin Component Respondent Responding Reiponse Not at all Somewhat Fairly well Very well Thorou^ly NoRespoiiie 



1. Degree based on 
profession's needs. 



needs are considered. 



comes presented at 
J outset of course. 

00 

if 

4. Degree learning ob- 
jectives clear and 
succinct. 



demonstrable. 



6. Degree training 
provides opportunities 
to practice what is 
taught. 



Participant 


107 


Very well 


- 


4 


14 


52 


37 


1 


Instructor 








14%} 


(10%) 


(49%) 


(35%) 






very wen 






6 

(15%) 

« 


20 

(50%) 


14 

(35%) 


4 


Participant 


107 


Somewhat/ 


33 


20 


32 


15 


7 


1 


Instructor 


(99%) 


Fairly wel! 


(31%) 


(19%) 


(30%) 


(14%) 


(7%) 




40 


Very well 




4 


10 


19 


7 


4 




(91%) 






(10%) 


(25%) 


(48%) 


(18%) 




Participant 


102 


Fairlv well 

1 Mil I ■ f f Wl 1 


21 


18 


24 


29 


10 


ft 

D 


Instructi ' 


(94%) 




(21%) 


(18%) 


(24%) 


(28%) 


(10%) 




37 


Very well 




7 


11 


13 


6 


7 




(84%) 






(19%) 


(30%) 


(35%) 


(16%) 




Participant 


102 


Fairly well 


14 


11 


31 


38 


8 


6 




(94%) 




(14%) 


(11%) 


(30%) 


(^7%) 


(8%) 




Instructor 


39 


Very well 




5 


12 


14 


8 


5 




(89%) 






(13%) 


(31%) 


(36%) 


(21%) 




Participant 


105 


Fairly well 


19 


10 


30 


31 


15 


3 




(97%) 




(18%) 


(10%) 


(29%) 


(30%) 


(14%) 




Instructor 


40 


Fairly well 




9 


16 


12 


3 


4 




(91%) 






(23%) 


(40%) 


(30%) 


(8%) 




Participant 


103 


Fairly well 


41 


6 


19 


25 


12 


5 


Instructor 


(95%) 
40 


Fairly well 


(40%) 
5 


(6%) 
7 


(18%) 
11 


(24%) 
11 


(12%) 
6 


4 




(91%) 




(13%) 


(18%) 


(28%) 


(28%) 


(15%) 



ERIC 



23a 



TABLE 10F (Continued) 



Category 

Type of Number of Median . Frequency of Response 

Training Design Component Respondent Responding Response Not at all Somewhat Fairly well Very well Thorouglily No Response 



00 
0) 

a 



uegree pariicipani 


Participant 


Q7 


KIa4 nt oil 

Not at all 


m 


1 1 
11 


r 

b 


9 


3 


11 


iniormBa oi progress. 




(yUTb) 




1/1%) 


(11%) 


(5%) 


(9%) 


(3%l 






Instructor 




oomewnat 


0 

0 


12 


7 


7 


urn 


10 






(77%) 




(24%) 


(35%) 


(21%) 


(21%) 




0. uggree oo|eciives, 


Participant 


lUO 


very wen 


1 


i 


17 


56 


29 


3 


maieriais, and activities 




ly/yb) 




11%) 


(2%) 


(16%) 


(53%) 


(28%) 




cohesive. 


Instructor 


40 


Very well 




4 


7 


20 


9 


4 






(91%) 






(10%) 


(18%) 


(50%) 


(23%) 




Q J)onrttc (MAhtiiA/ 1 
3. Ucyicc IceuDdCK/ 


rarticipant 


104 


tAJHigWI Id I 




13 


15 


410 


g 


A 

4 


evaluation useful. 


(96%) 




(45%) 


(13%) 


(14%) 


(22%) 


(6%) 






Instructor 


37 


Fairly well 


4 


12 


12 


6 


3 


7 






(84%) 




(11%) 


(32%) 


(32%) 


(16%) 


(8%) 




10. Degree of opportunity 


Participant 


106 


Very well 


13 


12 


22 


37 


22 


2 


to provide input to 




(98%) 




(12%) 


(11%) 


(21%) 


(35%) 


(21%) 




faculty. 




















11. Degree training up- 


Instructor 


38 


Very well 




fl 

1 


6 


14 


17 


6 


graded and modified 




(86%) 






(3%) 


(16%) 


(37%) 


(45%) 





in relation to current 
state of the art, 



Training Setting, 

Techniques and Faculty 

While participants and instructors are generally pleased with the setting of NCDA 
training programs, some aspects deserve attention. As reflected in Table IIF, some 
instructors and participants feet that class sizes may be too large and that there is in- 
sufficient contact between them outside the classroom. About 20 percent of the 
participants surveyed also believe there is a poor mixing of trainees, in terms of their 
roles and experience levels. 

Instructors express some concerns about the management of training events 
{Table 12F), especially in the areas of orientation and training of faculty, program 
modifications based on assessment, and to a lesser extent, appraisal of instructors' per- 
formance. 

Lectures emerge as the most frequently used method of teaching at NCDA, and 
most participants find them more useful than the variety of other techniques which are 
used by instructors from time to time (see Table 13F). 

The faculty receives a high rating by participants in terms of substantive knowl- 
edge, practical experience, and teaching ability {Table 14F), and a high percentage 
of instructors surveyed have participated in NCDA programs numerous times (Table 
15F). 



Additional 

Comments 

In response to a final question regarding overall strengths and weaknesses of 
NCDA, instructors and participants had more praise than criticism. Participants high- 
lighted the following strengths: gaining coliegial contacts and resources, expertise of 
the faculty, and practicality of the training. A fewer number of participants found the 
training impractical, the instructors lacking training expertise, too much time given to 
lecture, and participants too heterogeneous. A few instructors and a good many of the 
participants found the short seminars too brief or the Ca^^r Prosecutors Course too 
long. The only other demurrers from the instructors' generally complimentary remarks 
were the obsen/ations by a few that the participants were^ too heterogenous iii experi- 
ence and that the scope of some seminars was too narrow. 
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TABLE 11F 

ADEQUACY OF THE TRAINING SETTING (NCDA) 



Type Of Number/% 
Aspect Respondent Responding 

1. Similarity of participants' Participant 105 
roles/experience {97%) 

Instructor 43 

(98%) 

2. Sufficiently small classes Participant 105 

(97%) 

Instructor 44 

(100%) 

3. Sufficiently long courses Participant 106 

(98%) 

Instructor 43 

(98%) 

4. Match between partici- Participant 105 
pant needs and instruc- (97%) 
tor expertise Instructor 41 

(94%) 

5. Availability of instruc- Participant 99 
tors outside of class (92%) 

Instructor 37 

(84%) 

6. Sufficient number of Participant 105 
contact hours between (97%) 
participants and in- Instructor 40 
structors (91%) 

7. Reinforcement of be- Instructor 37 
havior/skills ^84%) 

8. Adequate support ser- Participant 98 
vices (91%) 

Instructor 41 

(94%) 

9. Adequate physical ac- Participant 106 
commodations (98%) 

Instructor 41 

(94%) 



Frequency of Response 

No 

Yes No Response 



•84 


21 


3 


(80%) 


(20%) 


• 


37 


6 


1 


(86%) 


(14%) 




60 


45 


3 


(57%) 


(43%) 




34 


10 




(77%) 


(23%) 




yi 


15 


2 




(14%) 




35 


8 


1 


(81%) 


(19%) 




95 


10 


3 


(90%) 


(10%) 




40 


1 


3 


(98%) 


(2%) 




70 


29 


9 


(71%) 


(29%) 




28 


9 


7 


(76%) 


(24%) 




83 


22 


3 


(79%) 


(21%) 




29 


11 


4 


(73%) 


(28%) 




33 


4 


7 


(89%) 


(11%) 




95 


3 


10 


(97%) 


(3%) 




38 


3 


3 


(93%) 


(7%) 




97 


9 


2 


(92%) 


(8%) 




41 




3 


(100%) 
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TABLE 12F 

MANAGEMENT OF TRAINING EVENTS (NCDA) 



Aspect 



CTP Instructor Response 



Number 
Responding 



N 



% 



Yes 



N 



No 



N 



% 



1. Clarity of lines of authority/ 38 
responsibility 

2. Adequacy of administrative 39 
procedures 

3. Instructors appraised of per- 42 
formance/expectations 

4. Instructors given orienta- 42 
tion/staff training 

5. Program modification based 38 
on assessment 



86% 



89% 



95% 



95% 



86% 



35 



37 



34 



22 



28 



92% 



95% 



81% 



52% 



74% 



8 



20 



10 



8% 



5% 



19% 



48% 



26% 
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TABLE 13F 
FREQUENCY OF METHODS USED (NCDA) 





Method 


Median Response 






CTP Participant 


CTP Instructor 


1. 


Lectures 


Most of the time 


Most of the time 


2. 


Discussion groups 


Some 


Often 


3. 


Panel discussions 


Not at ail 


Some 


A 


V^Se SIUQI6S 


N/A 


Some 


5. 


Role playing, simulation 


Not at all 


Some 


6. 


Videotaping 


N/A 


Often 


7. 


Moving pictures 


N/A 


Some 


8. 


On-the-job training 


N/A 


Not at ail 


9. 


Computer Assisted 
Instruction 


N/A 


Not at all 



ERIC 
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TABLE 14F 

PERCEIVED FACULTY COMPETENCE (NCDAI 



Frequency of Response . • 

Category Totally Minimally 
Number of Median Unknow- Knowledgs- Quite No 

Aspect Responding Response ledgeable able Adequate Knowledgeable Expert Response 



Degree of their sub- 103 Quite 
stantive knowledge. (95%) Knowledgeable 



(1%) 



10 

(10%) 



56 

(54%) 



36 

(35%) 



OQ 
Oi 

a 



Extensivenessof 
their practical ex- 
perience. 



105 

(97%) 



Quite 

Knowledgeable 



15 



42 

(40%) 



47 3 

(45%) 



Teaching ability. 



103 
(95%) 



Quite 

Knowledgeable 



(2%) 



29 

(28%) 



53 

(52%) 



19 



Number times 
called upon staff/ 
instructors since 
training for followup 
technical assistance. 



107 
(99%) 



Not at Once or 
all twice 



Not at all 



81 

(76%) 



15 

(14%) 



No 

Periodically Often Continually Response 



7 

(7%) 



3 

(3%) 



1 

(1%) 



1 



TABLE 15F 

FACULTY FOLLOWUP/CONT. RELATIONSHIP (NCDA) 
Called upon by CTP 

Number of Times Participants Invited back to Institute 



N % N % 

None 6 14 10 23 

I -5 11 26 22 51 
6-10 8 19 6 14 

II -25 7 17 5 12 
Greater than 25 10 24 0 0 



INTEREST IN TEACHING AGAIN 
YES (%) NO (%) 
98 2 
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7. NATIONAL COLLEGE OF CRIMINAL DEFENSE (LAWYERS 
AND PUBLIC DEFENDERS) 



(Editor's Note: Since the initiation of this Impact Evaluation, the name of 
the National College of Criminal Defense Lawyers and Public Defenders 
(NCCDLPD) has been changed to the National College of Criminal Defense 
(NCCD). These two titles have been used interchangeably throughout this 
report.) 

The National College of Criminal Defense-^Lawyers and Public Defenders 
(NCCDLPD), located at the Bates College of Law, University of Houston, was visited 
on December 4 and 5, 1978, by a two-person interview team. The site visit was initiat- 
ed with a meeting with the Dean to explain the purposes of the impact evaluation and 
the site visit, to review the schedule for the visitation, and to reach agreement on the 
staff personnel vyho were to be interviewed. A lengthy and detailed interview was then 
conducted with the Dean, utilizing the project director questionnaire and other ap- 
propriate data collection instruments as guides. The Dean also escorted the evaluators 
on a tour of the College's facilities, including those that are shared with the National 
College of District Attorneys. 

During the remainder of the visitation, interviews were conducted with, and data 
collected from, the following members of the College staff: 

• Associate Dean 

■ Director of Training 

■ Director of (Research and Publications 

■ Publications Editor 

• Administrative Assistant to the Dean 

■ Registrar 

r" 

■ Data Services Technician 

At the conclusion of the visit, a brief exit interview was conducted with the Dean. 
In addition, a draft of the findings contained in this report was submitted to the Dean 
for confirmation purposes. 

History of Institution 

The National College of Criminal Defense Lawyers and Public Defenders was 
founded in 1973 as a non-profit Texas corporation sponsored jointly by the American 
Bar Association, the National Association of Criminal Defense Lawyers, and the 
National Legal Aid and Defender Association. The bulk of its funding support comes 
from the Law Enforcement Assistance Administration. 
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The College is located on the campus of the Bates College of Law at the Univer- 
sity of Houston in office space shared with the National College of District Attorneys. 

The general purpose of the College is to train the nation's defense attorneys, with 
a priority of producing more highly trained and skilled defense lawyers for indigent 
defendants. 

The educational programs of the College consist of summer resident sessions at 
the University of Houston, national institutes or workshops held at regional locations, 
and defender management workshops. The College also sen/es as a resource center and 
produces two publications on a regular basts. More than 5,000 public defenders and 
private attorneys have attended the College's programs over the years. 

The College is headed by Dean John E, Ackenman, who has held that position on 
a full-time basis since 1974. The other key professional positions are Associate Dean, 
Director of Training, and Director of Research and Publications. At the time of our 
visit, we were informed by Dean Ackerman that the Associate Dean would be leaving 
at the end of 1978 and that the two other key positions would be abolished at the 
end of the LEAA grant year (3/31/79), 

The College is officially governed by a 12-member Board of Regents, four mem- 
bers representing each of the College's sponsoring organizations {identified above). The 
Board, which meets twice a year, establishes policy with guidance of the Dean. 

Comparisons of budget years are somewhat difficult because LEAA grant 
p<:iiods have varied in length. However, recent LEAA grants have been as follows: 

Dec. 1975-Jan. 1977-$500,000 
Feb. 1977-Mar, 1978-$424,041 
Apr. 1978-Mar, 1 979«$350,000 

Matching funds have been provided annually by the three sponsoring organiza- 
tions (a total of $12,963 each in the current fiscal year). In addition, the College is re- 
ceiving funds this year from the American College of Trial Lawyers ($10,000) and a 
private foundation ($2,500). 



Goals and Objectives 
of the Institution 

The goals and objectives of the National College of Criminal Defense Lawyers and 
Public Defenders, as stated by Dean Ackerman, are to improve the quality of represen- 
tation for the indigent defendant in the United States. A by-product is to improve the 
quality of representation for all defendants. 

These goals do not appear in any literature prepared by the College nor were they 
formally adopted by :he Board of Regents. They do appear, in essence, in the College's 
grant applications to LEAA. 
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Progmms 



Trends: Over the past five years, the content of the College's programs has moved 
from substantive law to strategy and techniques. Program format has been shifted from 
standard lectures to participatory methods. 

The College has been offering seven or eight LEAA-funded training programs each 
year for the past four years. The number of trainee contact hours has decreased 
steadily from 285 to 166 over that same period, but the number of program partici- 
pants has b&en up and down in a range of 864 to 1,147 during those four years. 

Despite the goal which places emphasis on improved representation for the 
indigent defendant, the College was considering a course in white collar crime defense 
because it is a popular subject and can possibly generate income for the College. It 
would be a non*LEAA grant course. 

Current: The LEAA grant application for the period April 1, 1978, to March 31, 
1979, lists the following program schedule: 

• / Three-day Defender Management Workshop 
m 5 Three-day National Institutes 

■ 2 Two-week Trial Practice Institutes (summer residential) 

The National Institutes, held on a regional basis around the country, deal with 
subjects of jury selection techniques, advanced cross-examination technic jcs (drug 
defenses), appellate advocacy, and forensic science. 

In addition to those LEAA-funded programs, the College presented a contract 
program in Bismark, North Dakota on the subject of Criminal Defense Tactics and 
Techniques. 

Needs Analysis: There is no formal process in place for assessing the needs of po* 
tential trainees. Dean Ackerman feels there is no need for such a formal process be- 
cause the College utilizes feedback from participants at training sessions, feedback 
from faculty, and the court experience of the College staff. He believes these are suffi- 
cient to identify the needs of practitioners in the field. 

Curriculum Development: The Curriculum Committee of the Board of Regents 
meets once a year to review and make decisions about courses, with input from Dean 
Ackerman. The Dean and Associate Dean share primary staff responsibility for design 
and development of curriculum. Most training materials are developed in-house. 

The two summer resident training programs utilized the original NITA model, 
but the Dean says it has been revised substantially. He has not seen NITA's material 
for a couple of years, so is not certain what materials they are using now. (The 
College's Director of Research and Publications told us he has never seen any material 
produced by NITA.) 
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NCCDLPD prefers the use of participatory and demonstration techniques in its 
training programs, including the use of videotapes and paid actors who sen/e as jurors 
In mock trials. 

Faculty Selection and Development: According to Dean Ackerman, the College 
has developed a list of about 75 individuals who are utilized as the core faculty. (The 
faculty records provided to us listed only 36.) The selection process starts with consid- 
eration of the nature of the program, and decision^ are then made about the best in- 
dividuals to present that particular subject. Recruitment of faculty is largely on the 
basis of personal references and/or going after respected experts. The College has com- 
piled a list of "the 30 best criminal lawyers in the country" and refers to it in seeking 
faculty. Dean Ackerman says that teaching at the College has become so professional- 
ly significant that he often receives communications from individuals who offer to 
serve on the faculty. 

According to the faculty files provided us, there are 36 faculty members identi- 
fied. Of the 36, three are female (8%) and three are black (8%). No other ethnic or 
racial minority representative was indicated. 

Dean Ackerman says the list of faculty provided us was the faculty for the 1978 
Summer Session "which is simply not representative of the total faculty involved in the 
College." He says the permanent faculty file contains 252 prospective faculty mem- 
bers. Of that number, there are, according to the Dean, 24 women (9.5%) and 11 
blacks (4%). No files were made available for our review covering either the 252 pro- 
spective faculty members or the 75 Dean Ackenman refers to as the "core" faculty. 

The Dean also spoke of his "substantial efforts" to recruit women and minorities 
as faculty. He claims that only recently have minority and women lawyers come into 
the business of defending criminal cases. Therefore, he maintains, the numbers with 
sufficient experience to have credibility at his College are substantially disproportion- 
ate to those which would be reflected by a census of bar member^ip. 

Because faculty members are busy individuals With heavy schedules, it has been 
difficult for the College to build in time for faculty preparation and development as 
part of the program format. However, the faculty always receives course materials and 
format information in advance of the program. For the two-week Trial Practice Insti- 
tutes, the faculty meets for three to six hours on the Sunday prior to the sessions in 
order to discuss preparations and presentations. During the two-^week course there is a 
faculty meeting each day, and Dean Ackerman monitors the instructors and obtains 
informal feedback from the trainees. Less time is devoted to faculty preparation at the 
three-day workshops, but monitoring and feedback provisions are employed. 

Permanent Staff: At the time of our site visit, the NCCDLPD had a permanent 
staff of 12. These included the following positions: 

• Dean 

• Associate Dean 
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Director of Training 



■ Director o f Research and Publications 

■ Publications Editor 

■ Registrar 

■ Administrative Assistant to the Dean 

■ Data Services Technician 

■ Secretary to the Associate Dean 

■ Secretary to the Directors 

■ Records Clerk 

■ Printer 

Exhibit A is the College's organization chart. We have added the footnotes. 

The Dean, Associate Dean, Director of Training, and Publications Editor are train- 
ed attorneys. The other members of the staff have varying professional backgrounds 
and training. The twelve-member staff, at the time of our visit, consisted of six males 
and SIX females, none of whom was a member of a racial or ethnic minority group. 

The salaries of all staff are funded 100 percent by the current LEAA grant. 

At the time of our site visit. Dean Ackerman informed us of important staff 
changes which were in the offing. He said that the Associate Dean would be leaving the 
College at the end of the year (1978) and that he would recruit a new Associate Dean. 
The responsibility lodged with that position has been largely curriculum development 
and materials development. 

The Dean also informed us that the Board of Regents had decided to abolish the 
positions of Director of Training and Director of Research and Development at the end 
of the current LEAA grant year. He noted that about 53 percent of the grant budget 
is devoted to personnel costs and that the Board suggested those costs be reduced by 
eliminating the two positions and employing a young lawyer as a program coordinator. 
That coordinator, along with the incumbent registrar, would assume most of the 
functions now performed by the Director of Training, which are far more concerned 
with arrangement of the physical details of the workshops and seminars than with 
training the faculty or the participants in the various programs. 

As for the elimination of the position of Director of Research and Publications, 
the Dean plans to shift publications responsibilities to the incumbent Publications 
Editor and the grants research/writing responsibilities to the administrative assistant, 
who would also assume the duties of office manager. 
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OFFICE 
MANAGER 



DEAN 



SECRETARY & 
ADMINISTRATIVE 
ASSISTANT 



ASSOCIATE 
DEAN 



SECRETARY 




DIRECTOR 
OF 

RESEARCH & 
PUBLICATIONS 






PUBLICATIONS 
EDITOR 

® 



REGISTRAR 



« The circled numbers, 1 through 6, denote the chain of command. 

I At the time of our visit, the position of Office IVIanager was vacant, and the duties were being fulfilled by the Director 
of Research and Publications. 



The net effect of these contemplated changes would be to reduce the size of 
the staff from 12 to 1 1 and to reduce the annual staff payroll by an amount not yet 
specified. 



Effectiveness 

Marketing of programs and selection of trainees: NCCDLPD will accept for its va- 
rious training programs virtually anyone who applies. For general admission students, 
admission is open on a first come- first serve basis to any licensed attorney to the extent 
of available space. For scholarship students (scholarships cover travel costs and living 
expenses), no less than 50 percent of the scholarships are awarded to private attorneys, 
based on their application statements as to the percentage of time they devote to de- 
fense of the indigent, a declaration of willingness to represent the indigent in the fu- 
ture, a statement of financial need, and some other relatively minor considerations. 
The names of scholarship students attending courses are forwarded to their local juris- 
dictions so that the courts will be aware of the students' pledge of willingness to repre- 
sent indigents. However, the College makes no effort to check back within a period of 
time to determine whether or not the student is keeping his pledge. 

All College programs are advertised via the institution's mailing list of 11,000 
public defender offices and private attorneys, pius advertisements in the ABA Journal. 

Monitoring and assessment of programs, faculty and trainees: As described previ- 
ously. Dean Ackerman monitors the programs and faculty presentations during the 
year. He does this by sitting in on programs and by talking with students informally. 
Students are also queried more formally by filling out a questionnaire in which they 
rate the programs, comment on the faculty, and generally describe the benefits of what 
they have learned. Only in the psychodrama program is there any checkback with stu- 
dents (six months later) to determine whether or not they are utilizing what they 
learned. 

Outside evaluations including key findings, responses and results: There have been 
two outside evaluations of the National College of Criminal Defense Lawyers and Pub- 
lic Defenders. The first was by Judge George Van Hoomissen of Portland, Oregon in 
1973-74, and the second by Norman Lefstein of the University of North Carolina 
School of Law in 1976. While both evaluations were generally positive, both noted 
the same area of concern . . . that private attorneys attending NCCDLPD programs had 
not in the past, and were unlikely to in the future, represented indigent defendants. 
Judge Van Hoomissen wrote in his evaluation: 

'7f is clear that the stated primary goal of the College is to upgrade the de- 
fense of the indigent accused. If this continues to be the primary justifica- 
tion for federal funding of the project, greater care must be taken in 
screening potential students in the future/' 

And in his evaluation report written two years later, Mr. Lefstein said: 

. . the evidence strongly suggests that a great many of the private attor- 
neys who attended the College's training programs during the second grant 
year provide littie, if any, representation of indigent defendants/' 
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We could find no evidence of any effective changes to address this concern. In 
fact, the decision of the College to initiate programs in the white collar crime area 
raises questions about the actual priority assigned to representation of the indigent 
defendant. We also found in our interviews with NCCDLPD training participants a 
number of private attorneys who have not represented indigents. Some attended the 
College on scholarships. 

Dean Ackerman takes issue with the two previous evaluations, calling their con- 
clusions ''unfair and unrelated to the fact existing at the time." He further states, 
"In the seven programs the College has presented beginning with the two 1978 aimmer 
sessions, 658 lawyers have been trained. Of that number 420 were public defenders 
who are, of course, totally involved in the defense of the indigent. In other words, 64% 
of those attorneys spend 100% of their time defending indigent persons. It is extremely 
difficult among the private bar to find criminal lawyers who do not devote at least 25% 
of their time to the defense of the indigent on either a court-appointed or pro bono 
basis." 



Other Institution Activities 

Research: The College engager in no research per se. It does, however, seek out 
and reprint government documents of interest to defense lawyers, then offer them for 
sale as a money-raising device. 

Publications: The College produces two publications on a regular basis annually. 
Criminal Defense, which includes articles and information concerning the criminal de- 
fense field, is published six times a year. The National Journal of Criminal Defense, a 
law review, is published twice a year. 

External Relations: NCCDLPD shares office space and certain equipment with the 
National College of District Attorneys. This arrangement undoubtedly reduces costs of 
both institutions to some degree. However, the two colleges do not attempt to coordi- 
nate their training programs for the purposes of sharing lecturers and other costs be- 
cause of space limitations at the University of Houston and because of what Dean 
Ackerman calls ideological differences. 

As for its relations with LEAA, the College would prefer more contact and on-site 
presence of LEAA representatives to observe what the College is doing. In most of its 
brochures and on the front cover of its publications, the College notes that its activities 
are funded through LEAA. 

The College has utilized some of the material prepared by the Court Practice 
Institute and by NITA, but ail material has been revised substantially, and Dean 
Ackerman says it is difficult to share such material. 

Although the Dean is not familiar with the current programs or training materials 
of NITA, he does identify it as the College's only real competitor. 

Alumni Development: Ongoing relations are maintained with alumni through the 
mailing list announcements and through Criminal Defense magazine, which is sent to 
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each member of the alumni group who pays a minimum annual fee of $15. Alumni are 
e\u> solicited for contributions, and the Dean wants to institute the practice of allow- 
ing the alumni to earmark the uses of their contributions. 



Program and Related Costs 

Budget History: Over the past three funding years, the College's spending levels 
hove varied with the decreases in LEAA funding support, but much of the decrease has 
been made up through increased tuition revenue. 

Revenue: 1976-77 1977>78 1978-79* 



Tuition/Fees $ 77,664 $105,735 $161,000 

LEAA Grant 500,000 424,041 350,000 

Sponsor Funding 55^40 47,222 52,963 

Other 284 323 7,492 

Total $633,188 $577,321 $571,455 

("^Current Fiscal Year's Revenues are estimates) 

Other steps taken by the College to address the reduction of the LEAA grant 
include the elimination of two staff positions (discussed previously) and the elimina- 
tion of rental payments to the University of Houston. (The $12,500 annual rent will 
be paid from non-grant funds.) 



Activities Analysis: During 1977, NCCbLPD conducted a total of seven instruc- 
tional programs ranging from three-day workshops to two-week residential seminars. 
Costs per student per day ranged from $111 for the three-day Defender Management 
Institute to $39 for two of the three-day workshops. 

No. of Hours of Cost 



Program 


Topic 


Students 


Instruction 


Total Costs 


Student/Dav 


2-weetc 
resident 


Trial 
Practices 


88 


68 


$50,087 


$ 56.92 


2-week 
resident 


Trial 
Practices 


96 


68 


$42,391 


$ 44.16 


3-dav 
workshop 


Evidence & 
Cross Exam. 


137 


15 


$18,671 


$ 45.43 


3-daY 
workshop 


Defender 
Management 


74 


15 


$24,639 


$1 10.99 
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Program 


Topic 


No. of 
Students 


Hours of 
Instruction 


Total Costs 


Cost 
Student/Dav 


3<iay 
workshop 


Evidence & 
Cross Exam. 


89 




$10,328 


$ 38.68 


3<lay 
workshop 


Forensic 
Science 


94 




$11,046 


$ 39.17 


3-d ay 
workshop 


Forensic 
Science 


117 




$15,245 


$ 43.43 



The higher costs for the Defender Management Workshop may be traced to the 
fact that it is the only three<iay program in which the College pays transportation and 
subsistence. In addition, and despite the inclusion of travel and living costs, enrollment 
has been relatively low. 

Since our site visit. Dean Ackerman has advised us that it is now his intention to 
cancel the Defender Management Workshop for 1979, and he blames LEAA for making 
such a decision necessary. He says the NCCDLPD has not been able to justify more 
than one such program per year due to the low level of interest among defender offices 
in the subject of management training. This low interest level, he says, is reflected by 
the low attendance at a program in which the College pays full transportation and sub- 
sistence. Now, Dean Ackerman says he has learned that the National Institute of Law 
Enforcement and Criminal Justice (a unit of LEAA) has awanded "a very large grant to 
another organization (University Research Corp,) to conduct seven defender manage- 
ment workshops throughout the country in the next 12 months." He says the manage- 
ment program is patterned "almost exactly" after those which NCCDLPD has been pre- 
senting, and that with the competition of seven such workshops annually, there is no 
wisdom to NCCDLPD continuing its workshop, especially given the low level of 
interest in the subject. 

Dean Ackerman says, "At no time was this College consulted regarding that 
project or made aware of it until it was an accomplished fact." 



Management 

Policy planning: The Board of Regents is technically responsible for planning and 
establishing the policies of the College, but the Dean plays a major role in shaping that 
policy. At the moment, the institution's policy planning appears to be largely influ- 
enced by the limitation of fund sources and the search for new sources. 

Budgeting: The annual budget process is initiated in December of each year by 
the Dean, with staff assistance. The present year's budget is used as the base for 
planning the next budget. 

Documents and procedures relating to administration: The College boasts a num- 
ber of signs of sound management and administration including an organization chart, 
policy and procedure manual, written accounting procedures, purchasing procedures, 
an up-to-date property inventory, written personnel procedures, current position 
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descriptions, and staff attendance records. It tacks a formal system for evaluating the 
performance of staff members on a regular basis. 

The College made a decision to purchase a computer and accompanying programs 
for $40,000 in order to modernize its mailing list maintenance and its accounting pro- 
cedures. The College felt that if it was going to increase its revenue from publications 
sales, it had to modemize its mailing list operation, which had been improved from file 
cards to addressograph. The computer provides far greater flexibility in using the 
11,000 unit mailing list and will also accommodate the growth that is anticipated. At 
the present time, the College is selling computer time to one non-profit organization 
and hopes eventually to share time with the National College of District Attorneys* 
However, the Dean of the latter institution has expressed no interest in purchasing 
computer time from NCCDLPD. 



Challenges and 

Future Directions 

Dean Ackerman says the major challenge of the College is money, and he feels 
he must spend too much of his time trying to generate new dollars through a variety 
of means. 

Despite the restrictions imposed by inadequate funding. Dean Ackerman would 
like the College to take its training programs "on the road" to Public Defenders' offices 
around the country, where the lawyers in those offices would become the faculty and 
run the sessions. He sees this approach— possibly using a mobile home as a traveling 
office— as one way to avoid the increasing costs of bringing traintes to sessions (air 
fares) and putting them up in hotels, etc. 

The Dean would also like to develop self-paced teaming modules on videotape, 
with instruction booklets and training materials. Lawyers, individually or in groups, 
could use the materials and progress at their own pace and on their own time 
schedules. 



Training Objectives/ 

Perceived Effectiveness 

Instructors at NCCDLPD'-feel they have been very successful in achieving the four 
priority objectives of the College, as they identify them (Table 8G). Instructors and 
participants alike tend to support the relevance of training and the multiplier effects 
displayed in Table 9G. (The only low rating concerns attempts by trainees to make 
organizational changes after completing training, but that more than likely reflects 
the status of the trainee in his/her organization.) 

Participants and instructors are also complimentary of the soundness of NCCD's 
training design, except for some criticism by participants about the lack of information 
on their progress during the training program (Table lOGX 
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TABLE 8G 

INSTRUCTORS' PERCEPTIONS OF LEARNING OBJECTIVES (NCCD) 



Objective 



1. Imprcve participants' pro- 
ficiency in their roles. 

2. Update and increase partici- 
*pants' substantive knowledge. 

3. Enable participants to intro- 
duce new techniques and pro- 
cedures in their systems. 

4. Enhance participants' under- 
standing of their roles. 

5. Increase communication and 
consultantion among profes- 
sional peers/colleagues. 

6. Enable participants to in- 
fluence/promote change 
among other courts person- 
nel. 



Perceived Achievement 

Average Category of 

Priority Rating Median 
(1-High;7-Low) Response 



2.30 



Very well 



2.65 



Very well 



2.70 



Very well 



3.23 



Very wefi 



3.64 



Fairly Well/ 
Very well 



5.22 



Somewhat 
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TABLE 9G 
MULTIPLIER EFFECTS (NCCD) 



Aspect 



Type of 
Respondent 



Nuniber/% 
Responding 



N 



% 



Frequency of Response 
YES NO No response 
N % N % N 



Wish to return to 
institute. 

Perception of par- 
ticipants' desire to 
return to institute. 

Perception of par- 
ticipants' actual 
return to institute. 

Share institute 
materials. 

Perception of par- 
ticipants' use of 
institute materials. 

Use institute materials. 

Attempt to make insti- 
tute-spurred per- 
sonal changes. 

Attempt to make insti- 
tute-spurred organi- 
zational changes. 

Recommend institute 
to others. 



Participant 101 



Instructor 

Instructor 
Participant 

Instructor 
Instructor 



Participant 
Participant 



24 

23 
103 

22 
24 



Participant 102 



101 
103 



97 93 92 8 8 3 

96 22 92 2 8 1 

92 18 78 5 22 2 

99 83 81 20 19 1 

88 20 91 2 9 3 

96 20 83 4 17 1 

98 85 83 17 17 2 

97 34 34 67 66 3 

99 94 91 9 9 1 



1 1 -96b 



TABLE 10G 
SOUNDNESS OF TRAINING DESIGN (NCCD) 



Category 

Type of Number of Median Frequency of Resjionse 

Training Design Component Respondent Responding Response Not at all Somewhat Fairly well Very well Thoroughly No Response 



1. 


Degree based on 
profession's needs. 


Participant 
Instructor 


103 

199%) 

23 

(92%) 


Thoroughly 
Very well 


2 

(2%) 


6 

(6%) 
1 

(4%) 


9 

(9%) 
1 

(4%) 


27 

(26%) 
15 

(65%) 


59 

(57%) 
6 

(26%) 


1 
2 


2, 


Degree individual's 
needs are considered. 


Participant 
Instructor 


104 

(100%) 

23 

(92%) 


Fairly well 
Very well 


22 

(21%) 
2 

(9%) 


16 

(15%) 
1 

(4%) 


15 

(14%) 
4 

(17%) 

• 


28 

(27%) 
12 

(52%) 


23 

(22%) 
4 

(17%) 


2 


3. 

T 

(0 
0) 
0 


Degree expected out- 
comes presented at 
outset of course. 


Participant 
Instructor 


99 

(95%) 
2i 

(84%) 


Fairly well 
Very well 


19 

(19%) 

1 

(5%) 


19 

(19%) 
3 

(14%) 


22 

(22%) 
5 

(24%) 


28 

(28%) 
9 

(43%) 


11 

(11%) 
3 

(14%) 


5 

4 


4. 


Degree learning ob- 
jectives clear and 
succinct 


Participant 
Instructor 


99 

(95%) 
21 

(84%) 


Fairly well 
Very well 


12 

(12%) 
1 

(5%) 


17 

(17%) 
2 

(10%) 


25 

(25%) 
6 

(29%) 


29 

(29%) 
6 

(29%) 


16 

(16%) 
6 

(29%) 


5 
4 


5. 


Degree objectives are 
demonstrable. 


Participant 
Instructor 


101 

(97%) 

23 

(92%) 


Very well 
Very well 


18 

(18%) 
4 

(17%) 


7 

(7%) 
2 

19%) 


15 

(15%) 
4 

(17%) 


36 

(36%) 
8 

(35%) 


25 

(25%) 
5 

(22%) 


3 
2 


6. 


Degree training 
provides opportunities 
to practice what is 
taught. 


Participant 
Instructor 


101 

(97%) 

22 

(88%) 


Very well 
Very well 


34 

(34%) 
1 

(5%) 


4 

(4%) 
1 

(5%) 


6 

(6%) 
4 

(18%) 


22 

(22%) 
8 

(36%) 


35 

(35%) 
8 

(36%) 


3 
3 
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TABLE 10G (Continued) 



Typeof Number ofMrdian Frequency of Response 

Training Design Component Respondent Responding Response Not at all Somewhat Fairly well Very well Thoroughly No Response 



7. Degree participant 


rarticipant 




Mnf of oil 

Noi aiaii 


ul 


Q 
0 


11 

1 1 


11) 

lU 


13 


5 


informed of progress. 




/ACO/ \ 






(0%; 


|l 1 /b| 


IIDTo) 


1 lO/Oj 




Instructor 


19 


Fairly well 


b 


J 


n 
i 


D 


Q 
0 






(76%) 


(26%l 


(16%) 


(11%) 


(32%| 


(16%) 




8. Degree objectives, 


rarticipant 


IU4 


very wen 


0 


4 


14 




HO 




materials, ano aciiviiies 




/innVni 

\ lUU/Q/ 




\07q| 


(4%) 


(14%! 


(39%) 


(41%) 




cohesive. 


In'tructor 


21 


Very well 




1 


5 


9 


6 


4 






(84%) 






(5%) 


(24%) 


(43%) 


(29%) 




uegree leeouacK/ 


rdruiipclill 


inn 


rdliiy Well/ 


33 


6 


11 


15 


35 


4 


evaluation useful. 




(96%) 


Very well 


(33%) 


(6%) 


(11%) 


(15%) 


(35%) 


3 




Instructor 


22 


Very well 


2 




5 


8 


7 






(88%) 


(9%) 




(23%) 


(36%1 


(32%) 




z 10. Degree of opportunity 


Participant 


103 


Very well 


6 


11 


22 


35 


29 


1 


to provide input to 




(99%) 




(6%) 


(11%) 


(21%) 


(34%) 


(28%) 




2 faculty. 




















11. Degree training up- 


Instructor 


21 


Very well 




1 


2 


10 


8 


4 


graded and modified 




(84%) 






(5%) 


(10%) 


(48%) 


(38%) 





in relation to current 
state of the art. 
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Training Setting. 

Techniques and Faculty 



The adequacy of the training setting gets high marks from those participants and 
instructors surveyed, except for some dissatisfaction with the large size of the classes 
(laoie iii,). Instructors have some concerns about management of the training events 
especiaUy m the orientation and training they are given (Table 12G). 

Lectures are identified by both participants and faculty as the most used teaching 
method with discussion groups and role playing or simulation utilized to a lesser ex- 
tent (Table 13G). Participants find the role playing/simulation more useful than the 
lectures, with discussion groups rated a distant third. 

Faculty receive an extremely high rating from participants in terms of substantive 
knowledge, practical experience and teaching ability (Table 14G), and the instructors 
tend to ba invited to return to the College (Table 150). 
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TABLE 11G 

ADEQUACY OF THE TRAINING SETTING (NCCD) 



Type Of Number/% 
Aspect Respondent Responding 



1. Similarity of participants' Participant 102 
roles/experience (98%) 

Instructor 24 

(96%) 

2. Sufficiently small classes Participant 101 

(97%) 

Instructor 24 

(96%) 

3. Sufficiently long courses Participant 100 

(96%) 

instructor 24 

(96%) 

4. Match between partici- Participant 102 
pant needs and instruc- (98%) 
tor expertise Instructor 23 

(92%) 

5. Availability of instruc- Participant 101 
tors outside of class (97%) 

Instructor 23 

(92%) 

6. Sufficient number of Participant 102 
contact hours between (98%) 
participants and in- Instructor 24 
structors (96%) 

7. Reinforcement of be- Instructor 20 
havior/skills (80%) 

8. Adequate support ser- Participant 97 
vices (93%) 

Instructor 24 

(96%) 

9. Adequate physical ac- Participant 102 
commodations (98%) 

Instructor 24 
(96%) 



Frequency of Response 

No 

Yes No Response 



87 


15 


2 


(85%) 


(15%) 




21 


3 


1 




• (1 1%) 




65 


36 


3 


(64%) 


(36%) 




20 


4 


1 








97 


3 


4 


(97%) 


(3%) 




19 


5 


1 




\£ 1 TOl 




91 


11 


2 


(89%) 


(11%) 




21 


2 


2 




\H70I 




81 


20 


3 


\€>\J7Qf 


V^W A}} 




22 


1 


2 


(98%) 


(4%) 




91 


11 


2 




(11%) 




17 


7 


1 


(71%) 


(29%) 




17 


3 


5 


(85%) 


(15%) 




92 


5 


7 


(95%) 


(5%) 




21 


3 


1 


(88%) 


(11%) 




96 


6 


2 


(94%) 


(6%) 




23 


1 


1 


(96%) 


(4%) 
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TABLE 12G 

MANAGEMENT OF TRAINING EVENTS (NCCD) 



CTP Instructor Retporoa 



Aspect 



1. Clarity of lines of authority/ 
responsibility 

2. Adequacy of administrative 
procedures 

3. Instructors appraised of per- 
formance/ expectati ons 

4. Instructors given orienta- 
tion/staff training 

5. Program modification based 
on assessment 



Number 
Responding 



N 

24 

24 
25 
24 
23 



_% 
96% 

96% 

100% 

96% 

92% 



Yes 



_N 

21 

22 
22 
16 
20 



J6 
88% 

92% 

88% 

67% 

87% 



No 



H 
3 

2 

3 

8 

3 



13% 
8% 
12% 
33% 
13% 
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TABLE 13G 
FREQUENCY OF METHODS USED (NCCD) 



Method Median Response 

CTP Participant CTP instructor 



1. 


Lectures 


Often 


Often 


2. 


Discussion groups 


Some 


Often 


3. 


Panel discussions 


Some 


Some 


4. 


Case studies 


N/A 


Some 


5. 


Role playing, simulation 


Some 


Often 


6. 


Videotaping 


N/A 


Often 


7. 


Moving pictures 


N/A 


Not at all 


8. 


On-the-job training 


N/A 


Not at all 


9. 


Computer Assisted 
Instruction 


N/A 


Not at all 
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TABLE 14G 



PERCEIVED FACULTY COMPETENCE INCCD) 



Aspect 



Frequency of Response 



Category Totally Minimally 
Number of Median Unknow- Knowledge- 



Quite 



No 

Expert Response 



Degree of their sub- 
stantive Icnowledge. 



104 Expert 
(100%) 



2 

12%) 



40 

(38%) 



61 

(59%) 



(0 

a 



Extensivenessof 
their practical ex- 
perience. 



Teaching ability, 



103 

(99%) 



104 

(100%) 



Expert 



Quite 
Know- 
ledgeable 



(2%) 



12%) 



2 

(2%) 



(1%) 



17 

(16%) 



29 

(28%) 



52 

(50%) 



71 

(69%) 



31 

(30%) 



Number times 
called upon staff/ 
instructors since 
training for followup 
technical assistance. 



103 
(99%) 



Not at all 



Not at Once or No 
ail twice Periodically Often Continually Response 



69 26 
(67%) (25%) 



7 

(7%) 



1 

(1%) 



1 



u 
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TABLE 15G 

FACULTY FOLLOWUP/CONT. RELATIONSHIP (IMCCD) 

Number of Timos 

None 
1 -5 
6-10 
11-25 

Greater than 25 

INTEREST IN TEACHING AGAIN 
YES (%) NO (%) 
92 8 
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Called upon by CTP 

Participants Invited back to institute 



N % N % 

2 9 3 15 

9 41 13 65 

2 9 2 10 

4 18 2 10 

5 23 0 0 



8. NATIONAL INSTITUTE FOR TRIAL ADVOCACY 



The site visit to the National Institute for Trial Advocacy (NITA) took place 
November 27 and 28, 1978. The team visiting the site consisted of three members. 
There was a kick-off meeting with the Director and the Development and Southeast 
Regional Director, Later, members of the survey team interviewed appropriate person- 
nel to complete the questionnaires. 



History of the Institution 

The National Institute for Trial Advocacy was founded in 1971, It came into 
being as a result of the recommendation of the American Bar Association's Special 
Task Force on Advocacy. That organization cited the tack of adequate training in 
advocacy as being "largely responsible for court congestion and for problems in the 
criminal field which are reaching constitutional dimensions," Spurred by the force 
of these statements, the American Bar Association joined with the American Trial 
Lawyers Association and the American College of Trial Lawyers as sponsors of NITA, 

Since its inception NITA has been developing as an organization, as a concep- 
tion, in its enrollment and itr <iervices. It is founded upon two simpia ideas of con- 
siderable power: (1) while the An:erican justice sy.stem upholds the adversarial system. 
It fails to adequately tn-i.i young lawyers to function within it; and (2) trial advocacy 
can be taught, but the process may be more expensive when compared with other 
forms of legal education because of the need to have students perform individual 
problems under the supervision of experienced trial lawyers. 

At the present time the Institute is conducting one national session at the Uni- 
versity of Colorado at Bould«r in the summer and fivt- regional sessions each year. 
The number of participants involved i the national sessions is 150, and the number in- 
volved in the regional sessions is 31U. Its top administrators, at the time of our visit 
were Professor Kenneth Broun, who directs the operations of the Institute at its offices 
in Chapel Hill, North Carolina; Professor Robert Oliphant, administrator-treasu. Twho 
IS located at the NITA office in Minneapolis; and Christine Opp, who is engaged in 
administration and directs the Southeast Regional program. The other administrators 
are: Professors Irving Younger and Abraham Ordover, co-directors of thp Northeast 
Regional; Professor Thomas F. Geraghty, director of the Midwest Regional; Professor 
Barbara Caulfield, director of the Northwest Regional, and Professor Charles Ares, 
director of the Southwest Regional, 
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The first NITA program in 1972 was a four-week training session involving 107 
lawyers and law school professors. The program was repeated in 1973 for 157 parti- 
cipants. The four-week sessions proved to be exhausting and the last week counter- 
productive so that the residential sessions were shortened to three weeks. As the 
program evolved so did the NITA teaching method. Team teaching was always a part 
of the program, utilizing five teachers: a judge, two experienced attorneys, a team 
leader and an assistant. The judge and the attorneys were volunteers who contributed 
their time but whose expenses were covered. The team leader and his assistant were 
both experienced law school professors. 

The teaching methodology involved heavy use of learning by doing, utilizing 
the technology of videotaping for critical evaluation of the students' performance 
of trial problems. The faculty was Instructed to lecture little, to critique, and when 
feasible, to demonstrate correct methods. However, the emphasis was on student 
performance. The readings, case materials, and booklets of instruction to faculty and 
students underscored these instructional methods. These teaching methods make the 
program unique. The first NITA director, A. Leo Levin of the University of Pennsyl- 
vania Law School, was a major contributor to the method. Professor Lc= in served 
until 1973. His successor. Professor Robert E. Keeton, of Harvard Law School, also 
made major contributions to the teaching methods which became part of the program. 
Professor Kenneth Broun, of the University of North Carolina Law School, suc- 
ceeded Professor Keeton in 1976 and through his vvritingc, contribution to texts, 
Leaching and administrative support, has continued the approach developed by the 
institute's founders. 

In 1975 NITA initiated a new series of regional programs while continuing with 
its three-week national session. The regional programs duplicated the national session 
but they were held in two segments of eight days, three to four months apart. It had 
been felt that many lawyers who needed the instruction that NITA could give them 
were not able to leave their firms and jobs for a three-week period. Regional ization 
made it possible for N!TA participants to attend closer to home. The programs have 
been held at the University of North Carolina Law School (for the Southeast Region), 
Cornell University Law School (Northeast Region), Northwestern University Law 
School {Midwest Region), and the University of Arizona College of Law {Southwest 
Region). The northwest regional was held at the University of Oregon. The split session 
offered academic benefits inasmuch a;3 it gave students a period of time to ponder 
what they had learned and what they still v^anted to learn the second eight-day session. 

In an attempt to reach more lawyers by offering equally intensive short programs, 
the Institute joined with the American Bar Association {ABA) in a consortium in 
1976. Professor Broun feels that this consortium has been a success and has "demon- 
strated that significant training in trial advocacy can occur in shorter time periods." 
He also notes that the cost of these programs is considerably less than the NITA tuition. 

Recently, in line with the expansion of its teaching techniques, the Institute 
has been developing and publishing a number of materials which form the basis for 
the NITA workshops. These include: 




11-99 



Kenneth S. Broun and James H. Seckinger, Problems in Trial Advocacy, 
{n.p.), National Institute for Trial Advocacy, 1977. 

Kenneth S. Broun and James H. Seckinger, Cases in Trial Advocacy , (p.p.), 
National Institute for Trial Advocacy, 1977. 

Michele Hermann, et al., I^fJTA Teachers Manual and Administrative Guide 
to 1977-78 Problems and Cases in Trial Advocacy , (n.p.). National Institute 
for Trial Advocacy, 1977. 

NITA consultants are currently working on additional case files and materials for 
use in the various NITA programs. These works assist in the repticability of the NITA 
program. They offer law schools and other programs the opportunity to duplicate 
the NITA techniques by giving them the materals that are used at NITA sessions. The 
Hermann book, in particular, provides the would-be teacher of trial advocacy with 
notes for critiquing the performance of students for each of the problems in the 
accompanying case and problem text, and for concrete approaches to the organization 
of classes. As a further supplement, in 1975 the Institute developed a series of video- 
tape lectures by Professor Younger (co-director of the northeast regional) on evidence. 
In addition, a series of audio-visual materials that can be sent to law schools or others 
concerned with the development of trial advocacy programs has now been developed 
by the Institute. 

In line with its goals and objectives, mi A has made an effort to train teachers 
m Its trial advocacy teaching methods, normally through a program after the national 
session. In April 1978 for the first time, the .'ns Jtute inaugurated a separate training 
session for teachers at Harvard University. In November 1978 a second session was 
held at the University of Arizona College of Law. Another session is planned for 
April 1979 at Harvard. Attendance : thase sessions ranged from 40 to 60 'vith nu- 
merous people on waiting lists for <jach program, which demonstrates the interest 
of law school faculty in NITA's teaching, methods. 

These activitit j suggest that one of the sub-goals of NITA has been the dissemi- 
nation of its methods in the law schools. Approximately 50 law schools have adopted 
NITA materials and educational approaches. To further this work, the Institute has 
been holding law school teacher training programs, such as the three-day program held 
after the 1976 Boulder session. The program wss attended by clinical law teachers and 
interested and concerned people fro.n local and state bar associations. Here they were 
exposed to the NITA approach to th., teaching of trial advocacy. The staff at NITA 
believes that the NITA program can be adopted for law school use at no greater cost to 
the school than most other courses and for significantly less cost than live client 
clinical education. Harvard, North Carolina, Illinois, Notre Dame, Hofstra, Cornell, 
Hastings, Northwestern and many other law schools have already instituted NITA 
programs. 

The Board of Trustees consists of judges from both the Federal and state benches 
and practicing attorneys. 
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The Board of Trustees has general administrative and financial responsibility 
for the national session, and the regionals are under the control of the regional direc- 
tors. Overall educational policy is set by the director. 

The current LEAA grant is $1 21 ,000. The annual N ITA budget is about $650,000, 
with the largest proportion of the budget coming from the tuition fees of $1,250 per 
student. Since many students cannot afford this, the Institute has a scholarship policy 
utilizing LEAA funds to help defray tuition costs for public defenders and prosecutors, 
to the extent that funds are available. NITA also helps defray some of its costs by 
selling publications, tapes and other teaching aids to law schools. It is the recipient of 
grants from the ABA Sec ion of Litigation, the ABA Section of Insurance, Negligence 
and Compensation, and the International Society of Barristers. 

Other organizations sponsoring NITA include the American Bar Endowment, 
the American College of Trial Lawyers, and private foundations. The Association 
of Trial L ;wyers of America, the Council on Legal Education for Professional Re- 
sponsibilites, and the Practising Law Institute have contributed financially in the 
past. 

In the presentation of its programs, NITA has affiliated with such organizations 
as the American College of Trial Lawyers, Section of Litigation of the American Bar 
Association, tfie Trial Lawyers Section of Bar Associations, and university law schools. 



Goals and Objectives 

of the Institution 

The National Institute for Trial Advocacy has been clear and steadfast in its goals: 
to contribute to the development of an adequately trained, professionally responsible 
trial bar, sufficient to serve the needs of justice in the United States; to train trial 
lawyers, and especially young lawyers, in advocacy; to develop methods and techniques 
for teaching and learning skills of the effective, professional trial advocate; to encour- 
age the teaching and learning of these skills, and to train teachers for service in law 
schools and continuing legai education programs. N!TA ts encouraging the adoption 
of its methods of training in trial advocacy thrc.igh the development of curricular 
materials, including the publication of books and manuals and th*» devel'-^ment and 
dissamination of videotapes* for self-study. These goals are stateo in a number of 
brochures, in the articles recently published by Professor Broun, and in NITA grant 
applications to the LEAA. 



Frogmns 

Trends/Current: NITA began its educational work with the establif'iment of 
a national program of four weeks' duration, held annually. After the first two national 
programs, it was decided that four weeks of such an intensive experience were too 
exhausting for students and faculty alike and that the final week was counterpro- 
du t^ /e. As a result, in 1974, the length of time for the national workshop was cut to 
three weeks. 
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Several factors led to the institution of regional workshops in 1975. As the pro- 
gram became known it was evident that even a three-week residential session entailed 
a sacrifice in terms of time and productivity that few agencies, especially smaller firms, 
could bear. Applicants from larger organizations rxceeded the availability of places 
in the national programs. The cost of travel was ano- er factor. As presently structured, 
the regional programs are identical in curriculum and time to the national program. 
However, they are broken into two parts, each part seven or eight days in 1 ?ngth. The 
programs are held in locations around the country: Chapel Hill, North Carolina; 
Ithaca, New York; Chicago, Illinois; Tucson, Arizona; and Eugene, Oregon. 

In 1976, to extent^ NITA's reach into the legal profession, the Institute joined 
with the ABA Consortiurj on Professional Education and with local bar groups to 
sponsor pilot and local state programs. These programs are shorter, running nine days 
rather than the fifteen to sixteen days of the n:itionai and regional workshops. The 
programs use videotapes of lectures given by Professor Irving Younger, a regional 
co-director of NITA's northeast regional. NITA is involved in the selection of at 
least one of the two teaching members of each program. NITA claims that the pro- 
grams are as intensive as the national and regional programs, although they cover less 
material. They do provide the student it is believed, a "significant training in trial ad- 
vocacy.... in shorter time periods." This type of program considerably reduces the cost 
to the student (in 1977 non-scholarship students were charged $1,000 for nationals 
and regionals, ano charges for the ABA-NITA programs were from $500 to $750). The 
current fee foi the regional is $1,250; the cost for ;'ie national program is the same. 

The ABA-NITA program is not directly affiliated with NITA, nor does the Insti- 
tute receive tuition from It. NITA is involved in a consulting capacity. A representative 
of the Institute goes to the state or locality that is interested in the program; the con- 
sultants find how many lawyers are interested in attending the program and how much 
time they can devote to it. They also determine what funding 's available. A NITA-type 
program is then tailored to these factors. The charge per student is frequently based 
upon, what the local organization itself can contribute. 

For 1P79 NITA plans a national 'program at Boulder, seven regional sessions and 
two special sessions for teachers. It will also continue to offer consultant assistance 
and support to the ABA-backed programs. 

In summary, five-year trends suggest the following: 

■ The focus of tfie institute on triai advocacy has remained the same 
throughout its brief history. 

• Implementation has involved a somewhat greater emphasis upon sep- 
arate programs for teacher training. 

* The programs have expanded considerably. In 1974 there was one 
naiioml program; today there are seven regional programs and two 
teacher training programs in addition to the national program. 
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• The contact hours between faculty and students reflect this expansion, 
increasing from a total of 176 hours in 1974 to a total of h084 in 1978. 

■ The total number of participants has grown steadily. In 1974 there 
were 172; in 1975 there were 272; 283 participated in 1976; in 1977 
there were 37a; and in 1978 there were 564. 

Needs Analysis and Curriculum Development: The general consensus about 
the state of the art of trial advocacy supports the need for trial advocacy education. 
Chief Justice Warren Burger has given this need wide publicity to the general discom- 
fort of the ABA and the law schools* On May 16th, at the opening of the American 
Law Institute meeting, he called for experiments by a limited number of law schools 
for one-year intensive special programs of advocacy training. 

Beyond accepting the wisdom of the Chief Justice, the Institute has developed 
a number of approaches to needs assessment: 

a. Entering lawyers have given their perception of their weaknesses as trial 
advocates. Exit questionnaires and follow-up reports suggest that these 
personal needs are being met. Professor Broun takes the evaluations into 
consideration in the preparation of the curriculum. Overall, they suggest 
that i>asic courtroom skills, oral presentations, and the ability to work 
wit^l lay and expert witnesses are needed. On the basis of evaluation of 
needs by the participants, time is budgeted for the presentation of certain 
topics. The needs assessment governs the time invested. Generally, these 
evaluations have noted that the development of cross examination skills 
are critical. The needs assessment impacts on the larger topic areas rather 
than on the more detailed aspects of trial advocacy. 

b. To measure the general perception of the need for trial advocacy training. 
Professor Broun called a meeting to discuss NITA educational objectives 
(December 1—3, 1978). The purpose of the conference was "to obtain 
the views of leading trial advocacy teachers with regard to what progress 
NITA has made in fulfilling ... objectives and what future steps NITA should 
take to better achieve its purposes." Presumably the inputs from the con- 
ference will help Professor Broun shape the curriculum of the Institute. 

c. The director is responsible for the overall coordination of the curriculum. 
A representative from the national office, either the director, development 
director, or administrator, attends each session, regional and national. 
The regional curriculum is set by the director who makes sure that the 
program remains the same from region to region. That is not to say that 
creative ideas do not come out of the regional seminars or that they do not 
provide an input into overall curriculum design. However, it does say that 
changes are deliberate and that the overall "purity" of the program design 
varies very little. 
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Although outside groups are not consulted in the development of the curriculum, 
individual consultants are used. Team leaders are involved in creating case files on 
specific topics such as civil anti-trust cases or narcotics cases. These files are then edited 
by the director and incorporated in the materials, depending on their usability in 
NITA programs. The faculty determines the materials to be used in the sessions. 

Faculty Selection and Development: There is no full-time teaching staff. The key 
administrators also teach part time. The rest of the faculty is selected by the director. 
The criteria include the potential faculty member's reputation and colleagues' knowl- 
edge of his or her abilities. The qualities sought include career experience as trial lawyers 
and facilitative personality with lawyers. The director and faculty are concerned that 
applicants have the ability to team teach. In the employment of minorities and women, 
NITA says it looks for faculty members who can be positive role models. Part of the 
faculty is paid and part is volunteer. IV mbers of the faculty who are professional 
teachers and who serve as team leaders for NITA sessions receive a stipend. The unpaid 
members of the faculty are the volunteer practitioners and judges. The volunteers 
are usually people with courtroom experience who also have experience with NITA. 
Judges are selected as part of the volunteer staff for their ability to team teach. A 
variety of judges - "soft" judges and "hard" judges - are sought in order to expose 
students to the realities of judicial contact. "Hard" lawyers are enlisted to give students 
exposure to the harassment they will get in actual courtroom situations. 

While these are th.-.* general, stated qualifications for the faculty, it was difficult 
for the site visitation group to ascertain with precision the number and qualifications 
of the actual faculty. There is a great deal o* regional independence regarding who is 
asked to teach and why. As a result, sometimes rather weak faculty members seem to 
be involved in the programs at the ragionai level. Inaccurate forms and files for the 
faculty made it difficult to evaluate this important area of the Institute. There were 
gaps in the curriculum vitae of some of the faculty members. A number of the files 
o-> faculty, the site visitation group was told, were no longer relevant since some of 
the individuals are not currently used as faculty, while other individuals mentioned 
as faculty members did not appear in the files. Therefore, no accurate file of faculty 
exists, and it was impossible for the evaluators to obtain reliable information about the 
characteristics of the actual faculty. As one example ^f this situation, when the evalua- 
tors raised questions about the low number of minority group members shown in the 
faculty files, eight names were presented orally. None of these names appeared in the 
faculty files, and all eight are purported to be black. Material on these eight faculty 
members with regard to experience and areas of expeitise was nonexist^jnt. 

Compensation for team leaders at all sessions is $3,750. National session team 
leaders receive a $1,250 expense allowance. Actual expenses are paid to regional team 
leaders. Compensation for all team assistants, both at the national and regional levels, 
is $2,000 {tor 1979). Team leaders at teacher training sessions receive $500. 

National and regional programs offer sessions to train the teachers and team 
leaders On the Saturday prioi to the beginning of a session, new faculty members 
and tejm leaders meet for two hours of preparatory worK. In addition, while the 
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sessions are in progress, team leaders observe the work of volunteer faculty. The 
Boulder and regional teanns are expected to nneet every day for at (east one-half hour 
to discuss emergent problems. They discuss the problems of individual students and 
remedial work that can be done with them. 

Prior to coming to the sessions faculty (and students) are expected to have read 
the case books and be ready to conduct the sessions. The actual preparation for the 
sessions can take from two days to two weeks. Approximately a week of the team 
leader's pre-session time is taken up with recruitment of the team. 

Permanent Staff: There is no iull-time teaching staff. At the time of our visitation, 
there were three secretarial/clerical persons in Chapel Hill and one in Minneapolis. In 
terms of permanent staffing, as opposed to full-time, the allocation of time is as fol- 
lows: Professor Broun, the director of the Institute, is employed as a three-quarter 
time director and devotes 20% of his time to administration, 10% to curriculum devel- 
opment, 20% to marketing, and 30% to conference management and participation. 
Christine A. Opp, who has been a practicing lawyer, is employed full-time by NITA as 
director of development and fund raising. She devotes 40% of her time to administra- 
tion, 10% to curriculum development, 20% to marketing, and 30% to conference man- 
agement and participation. Professor Robert Oliphant, who is with the Institute for 
one-quarter of his time, devotes 25% to marketing and 75% to conference management 
and participation. The four secretarial/clerical staff members divide their time as fol- 
lows: one full-time office manager/bookkeeper at Chapel Hill; one full-time clerical 
worker at Minneapolis; and two secretarial/clerical workers at Chapel Hi!L 



Effectiveness 

Marketing of Programs and Selection of Trainees: The programs are marketed 
through law journals, brochures, and announcements. In general, they are directed 
to lawyers in the first five years or so of law practice. Usually there are more appli- 
cants for the national session than places for them. 

Monitoring and Assessment of Programs, Faculty and Trainees: The director and 
development director oversee the programs; the regional directors monitor their 
faculties; and group leaders monitor assistants and volunteers. 

In addition, faculty is evaluated by participants on a five-point scaie fo" inter- 
action, critiques and explanations, assignments and exercises. The questionnaire 
includes a section in which the participant can recommend changes. Students similarly 
evaluate the section leaders. Students rate special lecturers and personnel on their 
integration into the program and their general usefulness. Students also rate tapes and 
audio-visual presentations. 

The evaluator. Dr. Russell Burris (see below), provides further performance data. 
He evaluates faculty on a ten-point summary scale. He also evaluates program aspects 
such as daily routine, the handling of special problems, full trial assignments, and the 
scheduling of full trial days. 
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Outside Evaluations: Evaluation has been part of the NITA program from its 
inception- Initially, evaluations were based on conferences between team leaders 
and the Board of Directors. Visitors were invited to the early sessions in 1972 and 
1973 to observe the teaching and make evaluative comments. Evaluative conferences 
were held among the director, associate dfrector, administrator and team leaders 
based on their observations. In 1974 NITA employed Dr. Russell Burris of the Univer- 
sity of Minnesota as evaluation director. Dr. Burris has designed an extremely provoca- 
tive and thoughtful approach for both the training process (see above) and the training 
outcome. In the former, he begins by asking participants to evaluate their own weak- 
nesses and periodically checks them during training to see whether the student still 
seos himself as weak in vital areas of trial advocacy. He has found that studerjts perceive 
themselves as gaining strength in areas of former weaknesses during the program. 
Students also give high ratings "to the overall program, and the program components 
give evidence to support \he conclusion that the students' expectations were met. 
These expectations tended to match tl\^ stated project objective." 

In addition, Burris had devised a system of external evaluation based on a careful 
definition of what constitutes strength in a trial lawyer. It is his belief that a teaching 
team comprised of a senior law professor, two experienced trial attorneys and a trial 
judge ought to be able to concur on these strengths. He therefore developed a checklist 
from their consensus which he used to evaluate students' progress, A post-evaluatior 
at ten months after the proyram is also used. 

Burris found that "Data do support the conclusion that improvement in the 
students' level of performance in trial skills was perceived by the participants, section 
leaders and references. Further, there is evidence that students gained methods which 
allowed them tc '^ontinue to improve their skills in the period following their partici- 
pation/' 



Other Institution 
Activities 

Research: The director of the Institute has been in touch with, and consulted 
with, numerous groups studying psychological and sociological patterns in the court- 
room and trial procedures. These groups have been encouraged to consult with NITA 
in their research and to use NITA programs as a research tool. 

External Relations: In terms of outreach or collaboration with other institu- 
tions there appears to be none, with the exception of the ABA mentioned above. 
The director vyouicl like to expand in this area. He says he is familiar with most major 
trial advocacy programs and the materials available in the field of trial advocacy. As 
for th:' ^ ationa) College of District Attorneys and ue National College of Criminal 
Defense Lawyers and Public Defenders, he feels the streng^jh*: of the two colleges in 
Houston are in pre-trial skills and office management, ar^'jas which NITA does not 
address. The Court Practice Institute is oaz of NiTA's CDmpoiitors. The two Houston 
colleges have curricular materials which compete with, Dut a' c complement, NITA's 
programs. 
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NITA mentions LEAA's support in their publications and brochures. Their bro- 
chure on learning and teaching trial advocacy, their gjneral information brochure, as 
well as the individual announcements of programs all mention LEAA as a primary 
funH'ng source. 

Alumni Development: A magazine, The Docket, is distributed to all former 
participants of NITA. The Advocacy Club is a fund-raising organization which is 
directed to NITA alumni as well as those interested in the development of trial advo- 
cacy in general. However, there is no alumni association per se. 

(Each of the eight training institutes was requested to supply lists of training par- 
ticipants for the purpose of selecting the case study sites and arranging interviews. The 
lists supplied by NITA proved to be incomplete and inaccurate to a far greater extent 
than any of the other institutes'. This ms/ be a further reflection of the administrative 
shortcomings discussed in connection with the maintenance of current faculty files. 
In addition, the situation raises questions about NITA's ability to maintain contact 
with \Xs alumni.) 



Program and 

Related Costs 

Buaget History: Since 1972 the LEAA has contributed the following dollar 
amounts to help defray the costs of the National Institute for Trial Advocacy: 

1972 $ 66,866 

1973 $ 70,200 

1974 $114,800 

1975 $126,890 

1976 $236,654 
n77 $260,198 
1978 $121,218 

Information regarding funding for fiscal years 1976, 1977, and 1978 to the date 
of the team's visit to NITA at Chapel Hill was difficult, if not impossible, to obtain. 
The bookkeeper/office manager was relatively new to her job; financial records prior 
to 1978 are in the Treasurer's Office in Minneapolis; and 1978 budget records were not 
even available in the Chapel Hill office, since a local data process service was working 
on them. This organization's turnaround time is such that the bookkeeper does not 
always receive prompt monthly reports, (The Director notes that the turnaround time 
for these reports is usually two weeks, which is an improvement over the time lag ex- 
perienced when NITA used the accounting services of the American Bar Association.) 

Despite the decline in funding by LEAA, NITA has managed to increase its 
revenue from ^575,445 in 1977 to $653,534 in 1978. Additional revenue was derived 
from contributions from the American College of Trial Lawyers ($26,000), The 
Advocacy Club ($35,224), increased sale of materials amounting to $100,070 and 
increased revenues from tuition (from $236,633 to $351,559). 
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However, due to the lack of year-to-date cost information, it is not possible to 
evaluate whether increased costs will be offset by the higher revenues. LEAA's decline 
In funding has impacted negatively on the relatively liberal scholarship policy NITA 
hc^ previously implemented. There is no doubt that the decline in LEAA funding will 
mean a direct decline in the nmount of scholarship aid which NITA will be able to give 
to public defenders and prosecutors. 

C St Analysis: The chart below presents the average cost per conference, the 
average number and cost per student, as well as the average cost per student per day. 



CONFERENCE STATISTICS 



Catogory 


Total No. 
Conferences* 


Average Cost 
per Conference 


Average No. 
Students 


Average Cost 
per Student 


Average Cost per 
Student per Day 


National 


1 


S139J34 


150 


$928 


$ 62 


Regional 


5* 


$ 42.862 


62 


$691 


$ 99 


Teachers 


2 


$ 10,270 


49 


$210 


$105 



"^Indicates conferences with available data. 



There is considerable difference in the costs per student for each region. The 
range is from $1,047 per student for the northwest regional to $689 for participants 
at the nnidwest regional. The number of students in attendance seems to be the crucial 
factor since costs of materials and support services were slightly higher in the midwest 
regional. The northwest regional had 44 students in attendance in 1978 compared to 
68 at the midwest regional. At the time of the evaluation team's visit, the northeast 
regional and the southeast regional could not be compared, since they either had not 
held their sessions or the figures were not available. However, the data for the first 
part of the programs showed the *ortheast regional operating on a $486 per participant 
basis while the southeast regional was costing $547 per participant. 

Two teacher training sessions were conducted. One session held at Harvard in 
April drew 60 participants and operated at $177 per participant; the November meeting 
in Tucson drew 38 participants, and, as a result, cost $262 per participant. One national 
session was conducted with 150 participants and a cost of $928 per participant. 

The figures for national and regional programs are based on costs provided by 
NITA. 



Management 

Policy Planning: Long-range planning, e.g., the Five^Year Plan, is conducted 
by the Director who consults with members of the board. The Director, Professor 



11-108 



Broun; the Treasurer/Administrator, Professor Oliphant; and Ms. Christine Opp, the 
Director of Development, meet once each quarter to discuss administrative matters 
and planning. Professor Oi phant also works as the planner and administrator of the 
national session. Regional session neads meet with the national organization for planning 
once each year. 

While there is no policy manual per se, there are some written policies, Simifarly, 
there is no formal organization chart, though there is a chart used for fund raising. 
(The national office has been working on the development of a policy manual. The 
director anticipates that it will be ready by spring, 1979J 

The Board of Trustees sets management and financial policy, while the director 
sets all programmatic and administrative policies. He delegates authority to the de- 
velopment director to coordinate regional programs with the Chape! Hill office. Re- 
gional directors are primarily responsible for their regions. 

Budgeting: This is a joint production of Professor Broun, Professor Oliphant, 
Ms. Opp, and Ms. Schoenbaum. Ms. Schoenbaum develops the cost figures and Pro- 
fessor Oliphant does a draft budget. From this. Professor Broun develops a budget 
for the LEAA. The budget is a simple and conservative line item without performance 
measures. A local accountant has computeiized all fiscal transactions. The system has 
been reco imend'?d by the auditors after having difficulty with the system used by the 
ABA. It has been ir. effect for approximately a year and a half, lin terms of purchr^sing 
procedures, one person— the office manager— is responsible. She is also responsible for 
the inventory of cifice supplies. The largest single purchasing item deals with printin^; 
and duplication of the books used in the sessions. All purchases are approved by the 
director or Ms. Opp. Auditing is handled by Price, Waterhouse and Co. which conducts 
an independent audit. 

ror property management and inventory there exists a simple inventory. The sale 
o^ books and videotapes is handled through the Minnesota office. Records of the sale 
of books and videotapes and the complimentary copies sent to law school faculty are 
available in the Chapel Hill and Minnesota offices. 

Written guidelines for payroll procedures call for the office manager to write the 
checks, which are countersigned. Check stubs are kept by the office and are transferred 
to the accountants. From this, a printout is made each month. 

A central checking account and four regional checking accounts ar^ maintained 
as w-gil as a savings account for the Advocacy Club only. All regionals have a checking 
account of $1,000. These are reimbursable. Minnesota has an account for $10 to 
$20,000. This is reimbursable through Chapel Hill. The office manager/bookkeeper 
handles all disbursements r reimoursements from the regional and national (Minn'?- 
sota-Boulder) account, as well as the payroll for vacuity and expenses. 

Ledgers consist of printouts which are bound on a monthly basis by the ac- 
counting firm, compared and controlled by the bookkeeper again . the checking 
account. All transactions are made by check. There arr no cash transactions (e.g.. 
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travel is not an item which is advanced). No one can get a cash advance and expenses 
are never advanced. All salaries are paid in arrears. Tuition is kept in the general ledger 
file and is cross-listed by region. 

NITA maintains a list of basic secretarial procedures; however, this is not In the 
form of a manual. Job descriptions are similarly not formalized in a manual. The job 
description for each employee is continuously updated as a result of conferences held 
quarterly between the director, the development director ard the office manager. 

Salary classifications are informal. Most secretarial staff have not been there for 
a full year. There is an informal merit increase system. Approximately 15% of salaries 
is in the form of fringe benefits. These include a health plan and life insurance. The 
health plan is contracted out. Each employee receives a $10,000 life Insurance policy. 
The Institute also purchases a retirement plan from TIAA/CREEF, after two full 
'ears of employment. These benefits are not in a manual but are on file in the office. 
Everyone other than secretaries {according to Ms. Opp, with whom the interview con- 
cerning personnel took place), is considered to be a consultant and is excluded from 
fringe benefits and the retirement plan. 

Personnel: Staff recruitment takes place through general advertisements and 
interviews. Some personnel are recruited through temporary off ice services and remain 
with the Institute. NlTA is an equal opportunity employer. 

Staff relations are informal. Evaluations are done by Ms. Opp and Professor 
Broun. There is no written ppUcy on staff attendance. They employ a flex-time con- 
cept which allows the staff to determine what is the best workipg schedule for them- 
selves and to employ their own self-restraints. 

All faculty and staff outside of Chapel Hill are considered consultants. The 
exception is Beverly Oliphant who does the secretarial work for the national programs. 
There are no written policies and procedures governing the work of the consultants. 
Past directors established policies allowing for maximum flexibility for regional direc- 
tors. There is guidance via a manual which is given to alt teachers with the teaching 
materials they must follow. Directors are given an administrative manual. In addition, 
there is an orientation program for all teachers. 

NITA employs a special consultant to assist with management and fund raising. 
The consultant, Arthur Frantzreb, of McLean, Virginia, is paid a flat consultant fee 
plus expenses for his work. Mr. Frantzreb is said to be ver-y generous with his time 
beyond his contractual obligations. 

For the recruitment of teachers, team leaders, and voliinteer faculty, sef. above. 
The supervision of the team falls upon the team leader who is in charge of controlling 
the team. He/she is >ware of the strengths and veak,/3sses of the team. Participants, 
students, and other team members frequently are not that cognizant of tht dynamics! 
Therefore, it falls to the team leader to direct the critiques - who critiques whom and 
when. The team leader is not supervised. iHowever each week the development direc- 
tor or the director makes a two- or three-day visit to a program. 
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Challenges and 

Future Directions 

When questioned aboutthe major challenges faced by the Institute, the director and 
development director had a number of interesting comments. One of the problems they 
see is that of being able to reach the number of trainees who need the help of NITA. 
Even with the large enrollment and the adoption of NITA's methods by law schools, 
there are far more lawyers who are graduating from law schools than could possibly be 
reached by these programs, to say nothing of those who graduated in the past. Another 
closely related problem is the lack of qualified team leaders— people who can direct 
NfTA regionals or NITA-type programs. Because of the small nu Tiber of qualified team 
leaders, the program has obvious limits in growth. The lack of money is another factor 
cited as inhibiting NITA's growth. The limitation of LEAA funding impacted upon the 
ability of NITA to give scholarships to lawyers who need aid. As a result, the program 
cannot reach some of the people who need NITA training the most. The lack of LEAA 
funding made NITA less able to give scholarship assistance; hence, it was harder to get 
public defenders and prosecutors to attend NITA sessions. At the same time, NITA pVo- 
gs'ams are expanding. NITA has been making a greater effort to develop in-house 
training programs, in the hope that the NITA program can be utilized by more people. 
The director is also concerned that replication efforts by other groups continue to be of 
the same quality as NITA's national and regional programs. 

While* not expecting to make major changes in the program, Professor Broun 
would like to hear, as a result of the national evaluation, how usable the trial techniques 
were in raising the quality of justice in the system. He would like to know what else 
they could be doing to increase this impact. He feels that closer coordination among 
schools supported by LEAA should an outcome of the impact study. He feels that 
NITA's teaching techniques could be transferred to other areas and facets of continuing 
legal education and would like to see these techniques adopted. 

In general, while convinced that NITA is "the finest post-graduate legal insti- 
tution in the country limited to trial work," he is receptive to change. This receptivity 
Is apparent in the plan which calls for continuous evaluation and makes possible the 
translation of these evaluations into programmatic changes. 



Training Ob/ectives/ 

Perceived Effectiveness 

NITA instructors believe they are very successful in achieving their four priority 
objectives, which are in accord with those of the institute {Table 8H). Participants and 
instructors rate the NITA programs as fully and very relevant, respectively. Instructors 
and participants generally give good ratings to the effectiveness of NITA, insofar as the 
multiplier effects shown in Table 9H are concerned. This seems to be con^i^Jstent with 
the fact that 92% of the participants recommend NITA to others, and with the very 
high ratings participants, and instructors, give to the soundness of NITA's training 
design {Table I OH). 
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TABLE 8H 

INSTRUCTORS' PERCEPTIONS OF LEARNING OBJECTIVES (NITA) 



Perceived Achievement 



Objective 



Average Category of 

Priority Rating Median 
(1-High; 7*Low) Response 



1 . I mprove participants' pro- 
ficiency in their roles. 

2. Enable participants to Intro- 
duce new techniques and pro- 
cedures in their systems. 

3. Update and increase partici- 
pants' substantive knowledge. 

4. Enhance participants' under- 
standing of their roles. 

5. Increase communication and 
consultation among profes- 
sional peers/colleagues. 

6. Enable participants to in- 
fluence/promote change 
among other courts person- 
nel. 



High; 
2,00 

2.63 

2,89 
3.11 
4.53 

5.59 



Very well 



Very well 



Very well 



Very well 



Fairly Well 



Somewhat 
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TABLE 9M 
N^'LTIPLIER EFFECTS (NITA) 



Aspect 



Type of 
Respondent 



Nutnber/% 
Responding 



N 



% 



Frequency of Response 
YES NO No response 
N % N % N 



Wish to return to 
institute. 

Perception of par- 
ticipants' desire to 
return to institute. 

Perception of par- 
ticipants' actual 
return to institute. 

Share institute 
materials. 

Perception of par- 
ticipants' use of 
institute materials. 

Use institute materials. 

Attempt to make insti- 
tute-spurred per- 
sonal changes. 

Attempt to make insti- 
tute-spurred organi- 
zational changes. 

Recommend institute 
to others. 



Participant 25 1 00 1 5 60 1 0 40 

Instructor 18 86 15 83 3 17 

Instructor 14 67 3 21 11 79 

Participant 25 1 00 1 6 64 9 36 

Instructor 18 86 14 78 4 22 

I nstructor 20 95 1 7 85 3 15 

Participant 24 96 22 92 2 8 

Participant 23 92 9 39 14 61 

Participant 25 100 23 92 2 8 



3 
1 
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TABLE lOH 
SOUNDNESS OF TRAINING DEW (NITA) 



Typeof Number ofMedian Fre^^xency of Response 

Training Design Component Respondent Responding Response Not at all Somewhat Fairly well Very well Thoroughly No Response 



1. 


Degree based on 


Participant 


25 


Very well 






7 


8 


10 






profession's needs, 




(100%) 








128%) 


(32%) 


(40%) 








Instructor 


20 


Thoroughly 


- 


- 


1 


8 


11 


1 








(95%) 








15%) 


i'»0%) 


(55%) 




2. 


Degree individual's 


Participant 


25 


Very well 


4 


3 


5 


10 


3 






needs are considered. 


(100%) 




(16%) 


(12%) 


(20%) 


(40%) 


(12%) 








Instructor 


21 


Very well 


- 


- 


1 


11 


9 










(100%) 








(5%) 


(52%) 


(43%) 




3. 


Degree expected out- 


Participant 


25 


Fairly well 


2 


8 


6 


6 


3 






comes presented at 




(100%) 




(8%) 


(32%) 


(24%) 


(24%) 


(12%) 




T 


outset of course, 


Instructor 


20 


Very well 




2 


4 


10 


4 


1 


at 






(95%) 






(10%) 


(20%) 


(50%) 


(20%) 




ft 

4. 


Degree learning ob- 


Participant 


24 


Fairly well/ 


3 


5 


4 


9 


3 


1 




jectives clear and 




(96%) 


Very well 


(13%) 


(21%) 


(17%) 


(37%) 


(12%) 






succinct. 


Instructor 


21 


Very well 






5 


8 


8 










(100%) 








(24%) 


(38%) 


(38%) 




5, 


Degree objectives are 


Participant 


25 


Very well 


1 




4 


10 


10 






demonstrable. 




(100%) 




(4%) 




(16%) 


(40%) 


(40%) 








Instructor 


21 


Very well 








11 


10 










i100%l 










(5?%) 


(48%) 




6, 


Degree training 


Participant 


25 


Thoroughly 




1 


2 


6 


16 






provides opportunities 




(100%) 






14%) 


(8%) 


(24%) 


(64%) 






to practice wliat is ' 


Instructor 


21 


Thoroughly 








5 


16 






taught. 




(100%) 










(24%) 


(76%) 





TABLE 10H IContinuedl 



Category 

Type of Number of Median Frequency of Response 

T raining Design Component Respondent Responding Response Not at all Somewi>at Fairly wr'! ^^erywel! Thoroughly No Response 

7. Degree participant Participant 25 Very well 1 13 8 12 - 
informed of progress. (100%) (4%) (4%) (12%) (32%| (48%) 

Instructor 20 Very well - - 2 9 9 1 

(95%) (10%) (45%) (45%) 

8. Degree objectives, Participant 25 Very well - 1 3 9 12 
materials, and activities (100%) (4%| (12%) (36%) (48%) 
cohesive. instructor 21 Thoroughly - - 3 6 12 - 

(100%) (14%) (29%) (57%) 

9. Degree feedback/ Participant 25 Thoroughly 1 - 4 5 15 - 
evaluation useful. (100%) (4%) (16%) (20%) (60%) 

Instructor 20 Vep/well/ - - 2 8 10 1 

(95%) Thoroughly (10%) (40%) (50%) 

^10. Degree of opportunity Participant 25 Very well 2 3 5 8 7 - 

i to provide input to (100%) (8%) (12%) (20%) (32%) (28%) 

^ faculty. 

11. Degree training up- Instructor 21 Very well - 1 2 9 9 - 

graded and modified (100%) (5%| (9%) (43%) (43%) 

in relation to current 
state of the art. 
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Training Setting. 

Techniques 
and Faculty 

While participants and instructors give generally high ratings to the adequacy of 
NITA training settings, they both have some negative reactions to the manner in which 
participants are mixed as to roles and experience {Table IIH). Instructors surveyed are 
very laudatory of the management of the training events {Table J2ffl. 

Participants and faculty agree that role playing/simulation, coupled with video- 
taping, constitutes the major teaching method at NITA. Although this is supplemented 
by some degreee of lectures and group discussions {Table 13H), participants over- 
whelmingly identify the role playing/simulation/videotaping method as the most use- 
ful. 

NITA's faculty is given a very high rating by participants, in terms of substantive 
knowledge., practical experience, and teaching ability {Table 14H), and the instructors 
tend to be invited back to N ITA training sessions {Table 15H). 



Other Comments 

Former NITA attendees were highly complimentary of the training they received. 
In response to a question regarding the strengths and weaknesses of the training, 
participants cited both positive and negative reactions. Among the strengths: the 
practicality of the training, faculty expertise, smooth organization, well-balanced 
methodology, use of training aids, and provision for feedback to participants were part 
of a strong positive consensus. Under weaknesses, the participants identified the fol- 
lowing: the course was too lengthy, the scope was inadequate, more variety was needed 
in the methods used, and the participants were uneven in experience and role types. 
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TABLE 11H 

ADEQUACY OF THE TRAINING SETTING {NITA) 



Frequency of Response 





Type Of 


Number/% 






No 


Aspect 


Respondent 


Responding 


Yes 


No 


Respoi 


1. olfnilanty of particip^ints 


• • 

Participant 


22 


10 


12 


3 


rolM/experience 




(88%) 


(45%) 


(55%) 






Instructor 


18 


9 


9 


3 






\oo%) 


(50%) 


(50%) 




2. Surfictentty small classes 


* 

Participant 


25 


24 


1 








(100%) 


(96%) 


(4%) 






Instructor 


20 


20 




1 






(95%) 


(100%) 






3. Sufficiantlv long courses 


Participant 


24 


21 


3 








(96%) 


(88%) 


(12%) 






Instructor 


20 


18 


2 


1 






(35%) 


(90%) 


(10%) 




4, Match between partici- 


• • 

Participant 


23 


18 


s 


2 


pant needs and instruc- 




(92%) 


(78%) 


(22%) 




tor expertise 


Instructor 


19 


19 




2 






(90%) 


(100%) 






o< MvanaDiiiiy oi insiruc- 


rarticipant 




24 


4 

1 




tors outside of class 




(100%) 


(96%) 


(4%) 






Instructor 


19 


19 




2 






(90%) 


(100%) 






Q. Sufficient number of 


Participant 


24 


22 


2 


1 


contact nours oetween 




{yD%/ 


(92%) 


inn/ \ 

(8%) 




participants ano in- 


instructor 


on 




4 
1 


4 

1 


structors 




(95%) 


(95%) 


(5%) 




7. Reinforcement of be- 


Instructor 


20 


20 




1 


navior/SKiiis 




(95%) 


(1(X)%) 






9. Adequate support ser- 


Participant 


23 


22 


1 


2 


Vices 






(96%) 


(4%) 






Instructor 


20 


17 


3 


1 






(95%) 


(85%) 


(15%) 




9. Adequate physical ac- 


Participant 


24 


24 




1 


commodations 




(96%) 


noo%) 








Instructor 


19 


19 




2 






(90%) 


(100%) 
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TABLE 12H 

MANAGEMENT OF TRAINING EVENTS (NITA) 



CTP Inrtructor Response 



A«pact 



1 . Clarity of lines of authority/ 
responsibility 

2. Adequacy of administrative 
procedures 

3. instructors appraised of per- 
forr;:ance/expectations 

4. Instructors given orienta- 
tion/staff training 

5. Program modificaticn based 
on assessment 



Number 
Responding 



N 



21 



20 



20 



20 



17 



% 



100% 



95% 



95% 



95% 



81% 



Yes 



N 



20 



19 



19 



18 



17 



% 



95% 



95% 



95% 



90% 



100% 



No 



1 



_% 
5% 

5% 

5% 

10% 
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TABLE 13H 
FREQUENCY OF METHODS USED (NITA) 





M«thod 


Median Response 






OTP Participant 


CTP Instructor 


1. 


LoctLirBS 


Some 


Some 


2 




Some 


• 

Some 


3 


Panfil HiscLicsinns 


Not at alt 


Not at all 


4. 


Case studies 


N/A 


Not at all 


5. 


Role playing, simulation 


Most of the time 


Most of the time 


6. 


Videotaping 


N/A 


Most of the time 


7. 


Moving pictures 


N/A 


Not at all 


8. 


On-the-job training 


N/A 


Not at all 


9. 


Computer Assisted 
Instruction 


N/A . 


Not at all 




II-112C 



2;^ 



TABLE 14H 



PERCEIVED FACULTY COMPETENCE (NITAl 



Aspect 



Frequency of Response 



Category Totally Minimally 
Number of Median Unknow- Knowledge- Quite No 

Responding Response iedgeable able Adequate Knowledgeable Expert Response 



Degree of their sub- 
stantive knowledge. 



23 Quite 
(92%) Knowledgeable 



4 8 11 
(17%) (35%) (48%) 



Extensiveness of 
their practical ex- 
perience. 



25 

(100%) 



Expert 



3 

(12%) 



(28%) 



15 

160%) 



25 

(100%) 



Knowledgeable 



4 



7 



(12%) (16%) 144%) (28%) 



Number times 
called upon staff/ 
instructors since 
training for followup 
technical assistance. 



25 



Not at Once or Nq 

^" twice Periodically Often Continually Response 

Notatall 20 4 1 - _ _ 
(80%) (16%) (4%) 



TABLE 15H 

FACULTY FOLLOWUP/CONT. RELATIONSHIP (NtTA) 
Called upon by CTP 

Number of Times Participants Invited back to Institute 



N % N % 

Nor^e 5 29 1 5 

I -5 4 23 11 55 
6- 10 4 23 7 35 

II - 25 2 12 0 0 
Greater than 25 2 12 1 5 



INTEREST IN TEACHING AGAIN 
YES (%) NO (%) 
100 0 
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OTHER TRAININGS-RELATED DATA 



Members of Institute 

Governing Boards 

Members of the governing boards of the eight training institutes were surveyed by 
means of a mail questionnaire. Their responses did not differ significantly from insti- 
tute to institute. 

In general, board members expressed the following views: 

• Most feel they influence policies relating to finance and program goals, 
with lesser influence on policies concerning staffing and curriculum; 

■ Most respondents believe that feedback from participants, normally 
through informal discussion, influences board policy decisions; 

■ Most board members say they are involved to some extent in curricu- 
lum development and selection of topics, but not in other institute 
activities. NCCD members say they are also involved in development of 
instructional methods, and AJC Board members claim involvement in 
a wide range of activities. 

No consensus exists across institute lines as to training objectives that are con- 
sidered essential. Most institutes, except AJC and NITA, feel that influencing coHegial 
communication is least important. 



Sanctioning and Support 
of Training 

In the case of all eight training institutes, participants said they had very strong 
desires to attend training; they support and endorse the concept; and they were en- 
couraged to a high degree by their employer/organization to attend. 

Instructors at the eight institutes concur in the above expressions and state that 
the institute-specific training has been endorsed by appropriate professional organiza- 
tions. 



Cost Comparisons 

Any examination of the comparative training costs of the eight CTP institutes 
must recognize the potential for disparities caused by the following factors: 
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Each institute calculated its own training program costs, using its own 
system; that is, travel /housing, staff salaries, overhead, etc., are handled 
in different ways by each institute; 



• No uniform system was used for charging overhead costs to specific 
training programs; 

• Each institute Y/as the sole source of data concerning attendance, 
instructional hours, and expenses associated with its training programs; 
and 

• 77?^ sophistication of the accounting systems utilized by the eight train- 
ing institutes varied substantially. 

All of the cost data which appear in the preceding sections dealing with the CTP 
institutes, as well as the data appearing in Table 16, were reviewed and verified by the 
respective institutes. 

In reviewing the table on Comparative Training costs, it is clear that the most ex- 
pensive program is the Appellate Judges Seminar program of the Institute of Judicial 
Administration. It is more than twice as costly as any other program, in terms of cost 
per student per day, and cost per student. This should not be surprising in view of the 
fact that the program covers all travel expense and subsistence for the senior and inter- 
mediate appellate judges who participate in the two-week resident programs at New 
York University. (LEAA funds, of course, are not used for travel and subsistence.) It 
must also be noted that the Appellate Judges Seminars received extremely high praise 
from the participants interviewed at the 12 case study sites. Further, a number of law 
professors, other trainers/educators, and supervisors rated the program as probably the 
best of the training vehicles available for appellate judges. 

The next most expensive training program, also a resident program in which travel 
and subsistence costs are included, is the Defender Management Workshop of the Na- 
tional College of Criminal Defense Lawyers and Public Defenders. It is because of a 
combination of the high costs, low attendance, and a competing program financed by 
LEAA that the NCCDLPD has decided to terminate the program. 

All of NlTA's training programs appear to be running at the higher levels of cost. 
Its national and regional programs rank among the highest in average cost per student. 
NlTA's regional and teachers workshops are among the most expensive in terms of 
average cost per student per day. 

The resident programs of the National Judicial College also appear to be in the 
high range of average cost per student, but lower in cost per student per day. ICM's 
3-5 day workshops are among the highest in average cost per student per day. 
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Both the National College of District Attorneys and the National College of Crim- 
inal Defense are in the low range of training costs. They share and use nearly identical 
space for their resident programs in Houston, but NCCDLPD's costs appear to be twice 
as high as NCDA's because NCDA separates its overhead costs from program costs. 

Another program which deserves mention for low training costs is the Appellate 
Judges Conference of the ABA. However, like NCDA, the "low costs" may be mislead- 
ing because of the way overhead costs are applied. 

Questions concerning the cost effectiveness of the eight training institutes will be 
discussed in Chapter III, Conclusions and Recommendations. 



Training Process 

Comparisons 

Based on our analysis of the eight training institutes and the survey of training 
participants, four elements of the training process emerge as keys to the effectiveness 
of the institutes. These are (1) needs identification or assessment, (2) faculty selection, 
(3) faculty preparation or training, and (4) monitoring and assessment of programs.' 
Based on the data we have collected and analyzed, the eight CTP institutes are rated in 
each category as shown below: 



Training 

Process 

Elements 


IJA 


AJC 


NJC 


AAJE 


ICM 


NCDA 


NCCD 


NITA 


Needs 
Assessment 


Strong 


Inadequate 


Strong 


inadequate 


Adequate 


Adequate 


Inadequate 


Strong 


Faculty 
Selection 


Strong 


Adequate 


Strong 


Adequate 


Adequate 


Adequate 


Adequate 


Adequate 


Faculty 
Preparation 


Adequate 


Adequate 


Strong 


Very Weak 


Inadequate 


Adequate 


Inadequate 


Adequate 


Monitoring and 
Assessment of 
Programs 


Strong 


Adequate 


Adequate 


Very Wealc 


Adequate 


Inadequate 


Adequate 


Strong 
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TABLE 16 
COMPARATIVE TRAINING COSTS 



Training 
Instituto 


Type of 
Training 
Program 


Total No, 
Of Events 


Avg. Cost 
Per Event 


Avg. No. 
Of Students 


Avg. uosi 
Per Student 


Avg. Cost 
rer oxuaenT 
Per Day 


IJA 


Appellate 
Judges 
Seminars 
(resident) 


2 


$ 46,964 


21.5 


$ 2,184 


$ 218 


AJC (ABA) 


Appellate 
Judges 
Seminar 
Series 


5 


$ 5,600 


34 


$ 164 


$ 41 


NJC 


Resident 
Extension 
Special & 
Innovative 
All (Average) 


36 
14 
1 


$ 30,501 
$ 829 
N.A. 


35 
65 
22 


$ 871 
$ 13 
N.A. 

$ 472 


$ 58 
$ 4 
N.A. 

$ 70 


AAJE 


National 
In-State 


,8 
10 


$ 5,804 
$ 6.496 


22 
90 


$ 264 
$ 72 


$ 61 
$ 34 


ICM 


Court Exec, 

Development 

(resident) 
Workshops, 

3-5 days, 

(resident) 

In-StatP 

III WLQ 

Workshops 


1 

19 
5 


$ 65,451 
$ 11.636 
$ 13,826 


19j 
31 
38 


$ 375 
$ 364 


q> Do 

$ 107 
$ 97 


NCDA 


Resident 
Short 
Course 
Trial 

Techniques 


4 
5 

2 


$ 18,258 
$ 5,232 

$ 6,150 


71 
94 

123 


$ 257 
$ 56 

$ 50 


$ 33 
$ 22 

$ 13 


NCCDLPD 


Resident 
Workshops, 

3-day 
Defender 

Mgmt. 

Workshop 


2 
4 

1 


$ 46,239 
$ 13,823 

$ 24,639 


92 
109 

74 


$ 503 
$ 127 

$ 333 


$ 50 
$ 42 

$ 111 


NITA 


National 
Regional 
Teachers 


1 
5 
2 


$139,134 
$ 42,862 
$ 10,270 


150 
62 
49 


$ 928 
$ 691 
$ 210 


$ 62 
$ 99 
$ 105 
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B. IMPACT FINDINGS 



The contents of this section of the report deal solely with data collected during 
or in connection with the field visits to the 12 case study sites, from the followino 
sources: " 

■ Interviews with CTP participants; 

■ Mall questionnaire responses from CTP participants; 

■ interviews with non-participant comparisons; and 

■ Interviews with supervisors and pe^ of CTP participants. 

Before being queried concerning training experiences, CTP participants and com- 
parisons were asked if there had been any significant changes in their court systems 
over the past five years. While there was no difference of consequence between the 
answers of the two groups {the vast majority acknowledged changes), appellate judges 
and prosecutors who had attended CTP institutes were more cognizant of systemic 
change (See roft/ei 7). 

This finding of systemic change is consistent with the contemporary trend nation- 
ally of improving and updating the justice system. Within the context of this atmos- 
phere of change, this study has sought to determine whether justice system training 
and continuing education have contributed to these changes, in what areas change has 
been influenced by training, and the extent of this influence. 

PERSONAL CHANGE AND ATTRIBUTIONS 

The CTP participants and comparison group members were asked whether they 
had experienced personal/professional changes over the past five years, in five specific 
categories: substantive knowledge, procedures they utilize, their personal skills, their 
use of resources, and the priorities they attach to their responsibilities. Where changes 
were acknowledged, the respondents were then asked to what they attributed those 
changes. The responses to these sets of questions are displayed by role group in Tables 
18A tfirough 25B. 

Appellate Judges 

Appellate judges of the intermediate appeals courts and the courts of last resort 
who had participated in CTP institutes claimed significantly more change in their per- 
sonal performances over the past five years than did those appellate judges in the com- 
parison group [Table 18A), These differences were most pronounced in the categories 
of procedures used, substantive knowledge, personal skills and use of r^ources. While 
some of the CTP participants cited training as the reason for their changes, both the 
participants and the coi.iparison respondents overwhelmingly attributed the changes 
to experience, changes in their court system, or "other" causes. Training and education 
received a low level of attribution. (See Table 18B). (It should be noted that only CTP 
participants attributed changes to training.) 
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TABLE 17 

PERCEPTIONS OF 15-YEAR) SYSTEMIC CHANGE 





CTP Participants 


Comparisons 




Number 
Responding 




Yes 




No 


Number 
Responding 




Yes 


Jjo_ 




Role Group 


N 


% 


N 


% 


N 


% 


N % 


N 


% 


N 


% 


Appellate Judges 


44 


96% 


40 


91% 


4 


9% 


23 


96% 


19 


83% 


4 


17% 


Trial Judges 


119 


98% 


98 


82% 


21 


18% 


66 


100% 


53 


80% 


13 


20% 


Private Attorneys 


45 


100% 


28 


62% 


17 


38% 


6 


100% 


4 


67% 


2 


33% 


Prosecutors 


94 


98% 


71 


76% 


23 


25% 


60 


100% 


38 


63% 


22 


37% 


Public Defenders 


71 


99% 


50 


70% 


21 


30% 


27 


96% 


19 


70% 


8 


30% 


Court Administrators 


30 


100% 


28 


93% 


2 


7% 


14 


100% 


13 


93% 


1 


7% 


Court Clerks 


12 


100% 


12 


100% 






5 


100% 


5 


100% 






Others 


43 


96% 


34 


79% 


9 


21% 


7 


100% 


3 


43% 


4 


57% 
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TABLE 18A 

PRESENCE OF CHANGE (APPELLATE JUDGES) 







CTPPartici 


panti 




. ComiMrisoni 


Type of Change 


Nuitibir 
notponoiM 


iiespondini; 
Yes 


Responding 
No 


Number 
Responding 


Responding 
Yes 


Responding 

Nq 


N % 


N 


% ■ 


N 


% 


N 




^1 


% 

58% 


N 
10 


% 

42% 


Substantive Knowledge 


45 98% 


34 


76% 


11 


24% 


24 


100% 


14 


Procedures 


45 98% 


32 


71% 


13 


29% 


23 


96% 


8 


35% 


IS 


65% 


Personal Skills 


45 98% 


26 


58% 


19 


42% 


22 


92% 


9 


40% 


13 


59% 


Resources 


45 98% 


28 


62% 


17 


38% 


23 


96% 


10 


44% 


13 


67% 


Priorities 


42 91% 


15 


36% 


27 


64% 


24 


100% 


8 


33% 


16 


67% 
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TABLE 18B 

AHRIBUTION OF CHANGE (APPELLATE JUDGES) 



Attribution 


Substantive Knowledge 


Procedures 


Personal Skills 


Resources 


Priorities 


ran* 




Psrt 

ran. 


LOITipi 


rari. 


tomp. 


rdtl. 


bomp. 


Part. Comp. 


1. Experience 


62% 


57% 


9% 


14% 


64% 


67% 


22% 


30% 


33% 25% 


2. System change 


6% 


7% 


28% 


43% 






19% 


30% 


25% 


3. New laws 


6% 


14% 


13% 


14% 










7% 13% 


4. Training 






12% 




16% 




7% 






5. Advice 






3% 






11% 


4% 




7% 


8. Education 


6% 


7% 








11% 








7. Reading 


6% 








4% 




.3% 


10% 




8. Other 


15% 


14% . 


34% 


29% 


16% 


11% 


44% 


30% 


53% 38% 
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Among those appellate judges who did attribute changes to training received at 
CTP institutes, IJA and AAJE, the latter for improvements in opinion writing, were 
most often credited. 



Trial Judges 

Among the trial judges surveyed at the 12 esse study sites, there was little dif- 
ference between the CTP participants and the comparison jurists in the degree of 
personal/professional changes they have experienced during the past five years. In three 
of the five categories of change, a slightly higher percentage of judges in the compari- 
son group claimed to have changed than did judges in the participant group [Table 
J^A). Training received more credit for these changes from participants, but overall, 
training was not a major factor (being cited by about 20 percent of the participant 
judges in three of the five change categories). Major attributions were experience and 
systemic changes. (See Table J 9B) . 

In all areas of change, NJC, which had the highest number of participants in the 
sample, was the institute most frequently mentioned as having influenced the changes 
made by participants. AAJE also received considerable mention in the change category 
of personal skills, largely for legal writing. Various state judicial conferences were also 
given credit in most categories of change. 

Private Attorneys 



There was a major inconsistency in the series of answers given by private attor- 
neys regarding personal changes over the past five years. Attorneys in the CTP partici- 
pant group cited changes to a substantially greater extent in the categories of substan- 
tive knowledge and personal skills, but more attorneys in the comparison group claimed 
change in procedures used and, to a lesser extent, in the use of resources. (See Table 
20 A). Training was cited as an important factor of change only in the area of personal 
skills, and by only 28 percent of the participant group. Experience was the overriding 
factor to which all private attorneys in both groups attributed their personal changes. 
{See Table 2 OB). 



Prosecutors 



A substantially higher percentage of prosecutors in the CTP participant group 
claimed changes in the procedures they use than did prosecutors in the comparison 
group. However, in all other categories of change, more comparison prosecutors 
claimed changes than did CTP participants {Table 21 A). Changes in all five categories 
were attributed to training by a small percentage of participant prosecutors (10 percent 
^? J. i^X?®"^' overall, experience on the job was the major reason cited. (See 
Table 2 IB). 



In all areas of change where training was the attribution, NCDA was cited most 
frequently. NITA was also mentioned, as were non-CTP institutes, most notably the 
Cornell Institute, the National Trial Lawyers Association, and statewide prosecutor 
training. 
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TABLE 19A 
PRESENCE OF CHANGE (TRIAL JUDGES) 





CTPParticipts 


Comparisons 


NurM 
Responding 


Responding 
Yes 


Responding 
No 


Number 
Responding 


Responding 
Yes 


Responding 
No 


Type of Change 


N % 


N % 


N 


% 


N % 


N 


% 


N 


% 


Substantive Knouiledge 


118 


98% 


86 


73% 


32 


27% 


65 


98% 


49 


75% 


16 


25% 


Procedures 


116 


96% 


70 


60% 


46 


40% 


63 


95% 


39 


62% 


24 


38% 


Personal Skills 


117 


97% 


75 


64% 


42 


36% 


65 


98% 


40 


62% 


25 


39% 


Resources 


117 


97% 


53 


45% 


64 


55% 


65 


98% 


35 


54% 


30 


46% 


Priorities 


115 


95% 


38 


33% 


77 


67% 


64 


97% 


21 


33% 


43 


67% 
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TABLE 198 

ATTRIBUTION OF CHANGE (TRIAL JUDGES) 



Attribution 


Substantive Knowledge 


Procedures 


Personal Skills 


Resources 


Priorities 


Part. 


Comp. 


Part. 


Comp. 


Part. 


Comp. 


Part 


Comp. 


Part. Comn 












1. Experience 


46% 


50% 


29% 


38% 


65% 


79% 




Ojtd 


35% 38% 


2. System chanoe 


2% 


13% 


9% 


22% 




3% 


20% 




12% 24% 


3. New laws 


8% 


4% 


15% 


14% 






Cm 




6% 


4. Training 


19% 


2% 


20% 




23% 


3% 


12% 




12% 5% 


5. Advice 






2% 


3% 




3% 


6% 




5% 


6. Education 


1% 


2% 




3% 


1% 




2% 




5% 


7. Reading 


11% 


17% 


2% 


3% 


6% 


5% 


4% 


3% 




8. Personal reasons 




2% 


5% 








8% 






9. Politics 






2% 


3% 






2% 






0. Other 


12% 


10% 


18% 


16% 


4% 


8% 


24% 


16% 


35% 24% 
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TABLE 20A 

PRESENCE OF CHANGE (PRIVATE ATTORNEYS) 





CTP Participants 


Comparisons 


Number 
Responding 


Responding 
Yes 


Responding 
No 


Number 
Responding 


Responding 
Yes 


Responding 
No 


Type of Change 


N % 


N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


Substantive Knowledge 


45 


100% 


36 


80% 


9 


20% 


6 


100% 


2 


33% 


4 


67% 


Procedures 


45 


100% 


26 


58% 


19 


42% 


6 


100% 


6 


100% 






Personal Sl(ill$ 


45 


100% 


37 


82% 


8 


18% 


6 


100% 


4 


67% 


2 


33% 


Resources 


45 


100% 


28 


62% 


17 


38% 


6 


100% 


4 


67% 


2 


33% 


Priorities 


45 


100% 


23 


51% 


22 


49% 


6 


100% 


3 


50% 


3 


50% 
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TABLE 20B 

AHRIBUTION OF CHANGE (PRIVATE AHORNEYS) 





Substantive Knowledge 


Procedures 


Personal Skiib 


Resources 


Prioritiei 


nuriuuiion 


Part 




rBTT. 


lomp. 


Part. 


Comp. 


Part 


Comp. 

75% 


Part. Comp. 
57% 100% 


1. Experience 


81% 


100% 


54% 


100% 


67% 


100% 


37% 


2. System change 






nn/ 

8% 








11% 




- 


0, newiaws 


3% 




4% 














4. Training 


3% 




8% 




28% 










5. Advice 






4% 








7% 






6. Education 


3% 












4% 






7. Reading 


6% 












7% 






8. Personal reasons 


3% 




15% 




3% 




4% 






9. Other 


3% 




8% 




3% 




30% 


25% 


44% 



31o 
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TABLE 21A 
PRESENCE OF CHANGE (PROSECUTORS) 





CTPPvticipantt 


Comporisom 




Number 
Ruponding 


Rispond 
Yet 


ling 


Respondins 
No 


Number 
Retponding 


Responding 
Yes 


Responding 
No 


TypeofChsnge 


N % 


N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


Substantive Knowledge 


95 89% 


80 


84% 


15 


16% 


59 


98% 


51 


86% 


8 


14% 


Procedures 


92 96% 


66 


72% 


26 


28% 


59 


98% 


34 


58% 


25 


42% 


Personal Skills 


95 99% 


82 


86% 


13 


14% 


60 


100% 




90% 


6 


10% 


Resources 


95 99% 


56 


59% 


39 


41% 


59 


98% 


37 


63% 


22 


37% 


Priorities 


94 98% 


50 


53% 


44 


47% 


60 


100% 


33 


55% 


27 


45% 
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TABLE 21B 

AHRIBUTION OF CHANGE (PROSECUTORS) 



Attribution 


Substantive Knowledge 


Procedures 


Personal Skills 


Resources 


Priorities 


Part. 


Comp. 


(jrt. 


Comp. 


Part 


Comp. 


Part 


Comp. 


Part Cnmn 










1. Experience 


64% 


74% 


44% 


49% 


78% 


83% 


28% 


51% 


38% 47% 


2. System change 


6% 




19% 


27% 


1% 


2% 


13% 


3% 


10% 6% 


3. New laws 


4% 


4% 


3% 


6% 










2% 3% 


4. Training 


10% 




16% 


3% 


14% 


4% 


15% 




13% 3% 


5. Advice 


1% 






■ 6% 






2% 






6. Education 




2% 


2% 






2% 




3% 




7. Reading 


4% 




3% 




1% 






5% 


2% 


8. Personal reasons 


1% 


2% 


2% 




1% 




2% 


3% 




9. New Management 






2% 








2% 




2% 


10. Other 


10% 


10% 


11% 


9% 


5% 


10% 


39% 


35% 


33% 41% 
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Public Defenders 

A greater percentage of defenders in the comparison group claimed changes in 
their performances over the past five years than did defenders in the CTP participant 
group, especially in three categories: procedures, personal skills and priorities. (See 
TfAle 22A). Although participant defenders cited training as the cause of change in 
four categories, only with respect to personal skills was the attribution of some signifi- 
cance (18 percent). Overall, experience on the job was cited as the major factor of 
change by defenders in both the participant and comparison groups {Table 22B). 

Where changes were attributed to training, NCCDLPD was most often mentioned. 
Court Administrators 

The number of court administrators surveyed at the 12 case study sites was gen- 
erally low, but it is nevertheless significant that substantially more in the CTP partici- 
pant group claimed personal changes in all five categories than did administrators in 
the comparison group {Table 23A). It is also noteworthy that training was cited as the 
reason for changes to a far greater extent by CTP participants in this role group than in 
any other (16 percent to 32 percent). Even so, experience on the job and changes in 
the court system outweighed training as change factors, (See Table 23B). Not surpris- 
ingly, I CM was the CTP institute most often mentioned by those citing training as a 
change cause. 

Court Oerks 

Tables 24A and 24B display the results of the surveys of court clerks in the 12 
case study sites. Changes were attributed to training by good percentages of partici- 
pants, but the overall sample of this role group is too small to analyze closely. When 
training was cited, I CM was the most oft-mentioned CTP institute. 

Others 

The "others" group is a potpourri of court-related individuals Including inves- 
tigators, law clerks, staff attorneys, etc. Because of the diversity of role groups, as well 
as training programs attended, analysis of this group is inappropriate. Tables 25 A and 
2SB are included for review. As can be seen, experience and system changes are again 
cited more often as causes of change. Only CTP participants mentioned training as at- 
tribution although comparisons attended as many training courses over the past five 
years. 



11-118 



TABLE 22A 

PRESENCE OF CHANGE {PUBLIC DEFENDERS) 





CTP Participants 


Comparisons 




Number 
Responding 


Responding 
Yes 


Responding 
No 


Number 
Responding 


Responding 
Yes 


Responding 
No 


Type of Change 


N 


% 


N 


% 


i 


% 


_N 


A 


1 


% 


N_ %_ 


Substantive Knowledge 


72 


100% 


56 


78% 


16 


22% 


28 


100% 


22 


79% 


6 21% 


Procedures 


72 


100% 


46 


64% 


26 


36% 


28 


100% 


21 


75% 


7 25% 


Personal Skills 


72 


100% 


63 


88% 


9 


13% 


28 


100% 


26 


93% 


2 7% 


Resources 


71 


99% 


44 


62% 


27 


38% 


28 


100% 


17 


61% 


11 39% 


Priorities 


71 


99% 


27 


38% 


44 


62% 


28 


100% 


15 


54% 


CO 



TABLE 22B 

AHRIBUTION OF CHANGE (PUBLIC DEFENDERS) 



Attribution 


Substantive Knowledge 


Procedures 


Personal Skills 


Resources 


Priorities 


rart. 


Comp. 


Part. 


Comp, 


Part 


Comp, 


Part. 


Comp. 


Part Comp. 










1. Experience 


75% 


64% 


60% 


48% 


79% 


84% 


38% 


25% 


27% 64% 


2. System change 






9% 


5% 


- 


- 


14% 


6% 


23% 7% 


3. New laws 


4% 


9% 


4% 


10% 


- 




- 


- 


- 


4. Training 


6% 




9% 




18% 


4% 


7% 


6% 


- 


5. Advice 




5% 










5% 


19% 




6. Education 




5% 




5% 












I Reading 


9% 


5% 




5% 




4% 




6% 




8. Personal reasons 


2% 




4% 








5% 


6% 




9. New Management 






2% 


5% 












0. Other 


6% 


14% 


11% 


24% 


3% 


8% 


31% 


31% 


50% 29% 
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TABLE 23A 

PRESENCE OF CHANGE (COURT ADMINISTRATORS) 



Type of Change 



Substantive Knowledge 
Procedures 
Personal Skills 
Resources 
Priorities 



CTP Participants 



Number 
Responding 



N 



% 



30 100% 



29 97% 



29 97% 



30 100% 



30 100% 



Responding 
Yes 

N % 



22 73% 



20 69% 



22 76% 



15 50% 



16 53% 



No 



8 27% 



9 31% 



15 50% 



14 47% 



Comparisons 



Number 



N % 



100% 



100% 



100% 



100% 



Yes 



N % 



10 71% 



6 42% 



10 71% 



6 43% 



3 23% 



Responding 
No 

N % 



4 29% 



8 57% 



4 29% 



8 57% 



10 77% 
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TABLE 23B 

AHRIBUTION OF CHANGE ICOURT ADMINISTRATORS) 



Attribution 



1. Experience 

2. System change 

3. New laws 



4. Training 



i s. 



CO 

a 6. 



7. New Management 

8. Professional 
Orgar 

9. Other 



Part Comp. 

55% 90% 

5% 

5% 

18% 10% 

5% 

5% 



9% 



Part. Comp. 

42% 17% 

16% 50% 
5% 
16% 

11% 17% 



11% 17% 



Personal Skills 
Part Comp. 

46% 75% 



32% 



5% 



13% 



18% 13% 



20% 



Resources 
Part Comp. 

40% 
20% 



fjfforities 
Part. Comp. 



44% 



33% 



25% 



33% 



7% 17% 



13% 83% 



31% 33% 
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TA8(i24^ 
PRESENCE OF CHAMQE jCOURTCLEfiKSj 







CTPPirtjfijpanti 


Conmriions 




NuRiiMr 
Rttpondini 


Responding 
Yes 


Responding 
No 


■ 

Number 
Responding 


Responding 
Yes 


Responding 
No 


TypsofChsngo 


N 


% 


N 


% 


N 


% 


N % 


N % 


N 




Suhttantive Knowledge 


12 


m 


10 


83% 


2 


17% 


6 


100% 


5 100% 






Procedures 


12 


100% 


9 


75% 


3 


25% 


4 


80% 


3 75% 


1 


25% 


Personal Skills 


12 


100% 


11 


92% 


1 


8% 


5 


100% 


4 80% 


1 


20% 


Resources 


12 


100% 


7 


58% 


5 


42% 


5 


100% 


2 40% 


3 


60% 


Priorities 


11 


92% 


4 


36% 


7 


64% 


5 


100% 


1 20% 


4 


80% 
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TABLE 24B 

AHRIBUTION OF CHANGE (COURT CLERKS) 



5 



Attributbn 



1. 

2. System change 

3. New laws 

4. Training 

5. Advice 

6. Education 

7. Reading 
a Otiier 




60% 80% 



10% 



10% 



20% 



Part. Comp. 



22% 33% 
22% 33% 
33% 
11% 
11% 

33% 



Personal Skills 
Part. Comp. 

60% 25% 



40% 25% 



50% 



Part. Comp. 



29% 50% 



29% 



14% 



Priorities 
Part Comp. 



25% 
25% 



100% 



14% 50% 



25% 



25% 
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TABLE 25A 
PRESENCE OF CHANGE (OTHERS) 





CTP Participants 


Comparisons 




Number 


Res 


ponding 


Responding 


Number 


Respond 


ing 




tponding 




Responding 




Yes 




No 


Responding 


Yes 






No 


Typ0 of Change 


N 


% 


N 


% 


N 


% 


N 


% 


N 


% 


N 


JL 


Substantive Knowledge 


42 


m 


34 


81% 


8 


19% 


7 


100% 


6 


85% 


1 


14% 


Procedures 


42 


m 


30 


71% 


12 


29% 


7 


100% 


4 


57% 


3 


43% 


Personal Skills 


42 


93% 


33 


79% 


9 


21% 


7 


100% 


5 


71% 


2 


29% 


Resources 


41 


91% 


25 


61% 


16 


39% 


6 


86% 


2 


33% 


4 


67% 


Priorities 


41 


91% 


18 


44% 


23 


56% 


7 


100% 


4 


57% 


3 


43% 



3;i 



ERIC 



TABLE 25B 
AHRIBUTION OF CHANGE (OTHERS) 



AIulDUIIOn 


Substantive Knowledge 


Procedures 


Personal Skills 


Resources 


Priorities 


Part. 


Comp, 


Part. 


Comp, 


Part. 


COITID 


Part. Comp. 


Part. Comp. 


1. Experience 


56% 


100% 


47% 




70% 


80% 


48% 


44% 25% 


2. System change 


3% 




3% 


75% 






4% 100% 


50% 


3. New laws 


6% 




7% 










11% 


4. Training 


15% 




13% 




21% 




22% 


11% 


5. Education 






7% 


25% 


3% 


20% 






6. Reading 


3% 
















7. Personal reasons 


6% 








3% 








8. Other 


12% 




23% 




3% 




26% 


33% 25% 



33 
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2MSTITUTE RELATIONSHIP 

TO AREAS OF CHANGE 



These data further delineate tine specific types of changes claimed as a result of 
institute training. In instances where changes were attributed to a specific CTP institute 
by the participants surveyed (in response to open-ended questions), the specific 
changes made and the frequency of the chariges were noted. These descriptions focus 
on training institutes rather than role groups. However, an assessment of each role 
group can be estimated because, for the most part, specific role groups correspond to 
specific institutes (See Table 1, Chapter I). 

INSTITUTE OF JUDICIAL ADMINISTRATION 

Two of the IJA participants surveyed (14.2 percent) indicated that changes in 
procedures were attributable to the training. The changes made were in terms of great- 
er awareness of issues and a change in the capacity to solve role- related problems. 

Similarly, two participants attributed changes made ir. personal skills to IJA. The 
Institute was credited with changing appellate opinion writing skills and with creating 
an emphasis on efficiency. 

Two participants indicated that changes in the use of resources were made be- 
cause of the Institute's training. Specifically cited were changes in the use of research 
resources and a change in the use of human resources through delegation of authority. 

In summary, the IJA program has influenced personal changes by its participants 
in three areas: procedures they use, personal skills, and use of resources. 

APPELLATE JUDGES' CONFERENCE (ABA) 

One participant (4-5 percent of the ABA's participants included in the study) 
indicated that a change had taken place in the area of substantive knowledge. Specifi- 
cally, the ABA program was cited as providing a better understanding of the criminal 
justice system in general. 

One participant of the ABA program indicated that, because of the training, a 
change was made in procedures, specinjcally in the use of appellate pre-argument 
settlement conferences. 

These are the only areas of change where the attribution was made to the AJC- 
ABA program. 



EKLC 



ii-n9 



NATIONAL JUDICIAL COLLEGE 



Eleven percent of the NJC participants included in this study indicated they have 
had a change in substantive knowledge due to the institute training. Most changes cited 
were in specific areas of law. 

Additionally, 1 1,3 percent of the NJC participants in the study cited the College 
as contributing to changes in procedures. The most frequently cited area was change in 
courtroom-related procedures, particularly techniques for handling jury trials, pretrial 
activities, and sentencing procedures- General administrative and management proce- 
dures, focusing on case calendaring and docketing, were also frequently cited. 

Similarly, nearly 10 percent of these participants reported changes in personal 
skills because of training. The most frequently cited areas of personal skill change were 
general courtroom-related skills and communication skills with peers and public. 

Some NJC participants surveyed also attributed changes in use of resources to 
training. This attribution was made by 5.2 percent of the respondents, mostly in the 
area of resources for research. 

The area of least attribution of change to NJC training is priorities. Only 3.4 per- 
cent of the participants attributed changes in this area to training. These changes 
were in priorities related to courtroom activities, research priorities, communication 
with defendants, and emphasis on justice and humanitarianism. 

In summary, NJC's greatest impact on its participants, in terms of helping them 
make changes in the way they work, was in the areas of substantive knowledge and 
procedures: 

AMERICAN ACADEMY OF JUDICIAL EDUCATION 

Three persons or 9.3 percent of the AAJE participants surveyed indicated that 
they experienced changes in substantive knowledge as a result of training by AAJE. 
This was primarily with respect to general understanding of the law. 

Furthermore, 6.2 percent of the AAJE participants surveyed indicated that they 
have made changes in procedures because of institute training. Two specific areas were 
cited: 

■ Knowledge of state and local court procedures; and 

■ Greater emphasis on procedures related to timeliness. 

More than 28 percent of the AAJE participants cited changes in personal skills 
because of the training. Most of these changes were in the area of opinion/legal writing 
skills. However, changes in self-confidence, changes in communication skills, and 
changes in work emphasis toward efficiency were also noted. 
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Ncne of the AAJE participants surveyed cited institute training as contributing 
to changes in the use of resources. 

Only one participant, 3.1 percent of those surveyed, attributed a change In pri- 
orities, specifically a priority related to plea bargaining, to the AAJE program. 

INSTITUTE FOR COURT MANAGEMENT 

Of the I CM participants surveyed, 11.4 percent indicated that changes they have 
made in the area of substantive knowledge were due to I CM training. For the most 
part, these changes were in understanding of particular areas of the law. 

Similarly, 9.8 percent of the participants indicated that they have made changes 
in procedures because of ICM training. The area cited most frequently wes, not sur* 
prisingly, changes in administration and management procedures. Usage of computers 
and new technologies are two of the specific areas cited. 

The greatest impact of ICM training on personal changes is in the area of personal 
skills. Nearly 23 percent of the ICM participants indicated that the training contributed 
to changes in this area, particularly in terms of skills related to management and 
administration. 

A number of attributions to training were also made by ICM participants with 
respect to changes in use of resources. Eleven percent of the ICM participants surveyed 
made this attributioa The changes cited fall principally into two areas: general use of 
resources, and changes in the use of human resources. 

Few ICM participants attributed changes in priorities to the training. Of those 
who did (1 percent), priorities for caseload management were cited. 

In summary, several ICM participants surveyed indicated that training contributed 
to their making personal changes In several areas, particuiariy related to skills and 
knowledge in administration and management 

NATIONAL COLLEGE OF DISTRIC T ATTORNEYS 

Of the 108 NCDA participants included in the study, 5.5 percent cited changes 
made in their substantive knowledge due to training provided by NCDA. Primarily, 
these changes were in substantive areas of the law (i.e., criminal law and narcotics), 
and in knowledge of courtroom-related activity. 

Additionally, 4.6 percent of the NCDA participants surveyed indicated that they 
have made changes in procedures directly because of the College's training. Five areas 
of procedural change were cited: 
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Change in capacity to solve role-ralated problems; 



• Knowledge of procedures related to organized crime; 

o Understanding of administrative and management procedures; 

• Greater understanding of communications, particularly as related to 
delegation of responsibility; and 

• Emphasis on procedures for increased efficiency. 

With respect to changes in personal skills, 4.6 percent of the NCDA participants 
attributing the changes to the NCDA program cited a number of areas, most notably 
changes in courtroom-related skills. Also cited were changes in general personal skills, 
skill in witness preparatioa and increased communication skills with peers. 

A slightly higher percentage (5.5 percent) responded that ecific changes in use 
of resources were attributable to NCDA. Most of these were in the area of use of 
human resources, as well as developing non-legal resources. 

An equal percentage (5.5 percent) responded that their priorities have changed 
because of the training, particularly in administrative and management matters. 

In sum, NCDA was reported to have contributed to changes in all five get:-5ral 
categories. Nearly equivalent percentages of participants made the attribution to 
training in these areas, 4.6 percent to 5.5 percent 

NATIONAL COLLEGE OF CRIMINAL DEFENSE 
(LAWYERS AND PUBLIC DEFENDERS) 

Approximately 4 percent of the NCCD participants surveyed indicated that 
changes have been made in their substantive knowledge directly because of the NCCD 
program. Most of the changes cited (50 percent) were in knowledge related to court- 
room activity. 

Similarly, 6.7 percent of these participants attributed changes made in procedures 
to NCCD training, most notably in the area of capacity to understand role-related 
problems. 

The most significant area where NCCDtraining was cited as contributing to per- 
sonal changes is in the area of personal skills. Twenty- two percent of the NCCD partici- 
pants surveyed made this attributioa This training was credited with influencing per- 
sonal changes in courtroom-related skills, particulariy examination of witnesses. 
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In terms of changes in use of resources, 7.6 percent of the NCCD participants sur- 
veyed credited the training with contributing to the change The specific areas where 
training did mske a difference are general use of resources, and resources related to 
evidence and witnesses. Less noted were resources for case preparation, resources for 
greater efficiency, and use of human resources. 

Only 1 percent of the NCCD participants in the study attributed changes in 
priorities to the training. Specifically, priorities related to case preparation were cited. 

In summary, NCCD training was cited by participants as contributing to personal 
changes. The most notable area of personal change because of training is personal 
skills, where 22 percent of the NCCD participants surveyed cited this attribution. 

NATIONAL INSTITUTE FOR TRIAL ADVOCACY 

Twenty percent of the NITA participants surveyed indicated that they have made 
changes in procedures because of tfie training. These changes fall into two main cate- 
gories, courtroom-related procedures, and case preparation procedures. 

Similarly, a high percentage of the participants surveyed (32 percent) cited 
changes in personal skills because of NITA training. Not surprisingly, most of the 
changes cited were in the area of courtroom*reiated skills and opening/closing argu- 
ments. 

Only one NITA participant cited a change in the use of resources because of the 
training. However, the change cited was, again, related to courtroom activity (i.e., 
resources related to evidence). 

Two of the participants surveyed (8 percent) indicated that NITA contributed to 
changes in priorities. The changes made were in general work emphasis, such as empha- 
sis on particular types of cases. 

in sumnjary, NITA training has been credited with influencing changes in its 
participants in several areas, procedures, personal skills, resources, and priorities 
More specifically, however, NITA seems to have the greatest influence on courtroom- 
related behavior. 



Change /A itribution 
Summary 

A number of respondents, both participant and comparison, in all role groups 
indicated that they have made changes in the way they do their work over the last 
five years. Most changes seem to have occurred in the areas of substantive knowledge, 
procedures, or personal skills for all role groups, participant and comparison. The area 
of least change for alt role groups, participant and comparison, was priorities. 
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While the attributions for these changes vary among the various role groups 
and between the participant and comparison groups, two trends are clear. For most 
changes made by members of all role groups, participant and comparison, experience 
is a key factor leading to changes. Attributions for changes to training (ail training) 
were made much more frequently by participants than comparison individuals This 
trend is particularly significant when compared to the amount of training attended by 
these individuals in the various role groups. Furthermore, when the attributions to 
training were made by participants, CTP institutes were cited most frequently. 

The types of changes made most often by participants of the institutes were in 
the areas of substantive knowledge and personal skills. Additionally, changes in pro- 
cedures were frequently cited as being made because of training. 

Those CTP institutes most frequently cited by those participants who did attri- 
bute changes to training were as follows, for each category or type of change: 

Substantive knowledge I CM 

NJC 
AAJE 

Procedures NITA 

IJA 
NJC 
I CM 

Personal Skills NITA 

AAJE 
I CM 
NCCD 

Use of Resources IJA 

iCM 

Priorities NITA 

NCDA 

DIRECT RELATIONSHIPS BETWEEN TRAINING AND ATTEMPTED CHANGE 



During the survey of CTP participants in the 12 case study sites, they were asked 
a series of questions about the institute's) they had attended and the training they 
had received. Then, the participants were asked if they had, as a result of the CTP 
training, attempted to make personal or organizational changes, and if so, to specify 
those changes. 

Tables 26 and 27 reflect the percentages of the total survey sample who (for each 
CTP institute) reported attempts to make personal and/or organizational changes be- 
cause of what they had learned. (N/A indicates that there were no participants in the 
particular role group in the survey). 
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As indicated in Table 26, "at least 50 percent of all role groups surveyed for 
each OTP institute said they attempted to make personal changes because of insti- 
tute training. A nnuch lower percentage of the participants of each institute reported 
that they have actually made personal changes because of training received, when they 
were asked about changes in general over the past five years. 

Table 26 indicates that six of the institutes (NITA, NCCDLPD, IJA, NCDA, NJC 
and AAJE) have a fairly comparable and effective rate of success in encouraging their 
participants to make personal changes based on training received. Two institutes, 
AJC/ABA and I CM, seem to be far less effective in that regard, based on the survey 
results. 

The percentage of participants who attempted to make organizational changes be- 
cause of CTP institute training varies considerably from role group to role group and 
institute to institute, as reflected by Table 27. None of the institutes appears to have 
been overly successful in this regard, but ICM seems to reflect the most success. 

Tables 28A through 3SB reflect the number of respondents in each role group, for 
each CTP institute, who cited specific attempts at personal and organizational changes 
as a result of training. These data supplement the training assessment data reflected in 
Tables 26 and 27. 
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TABLE 26 

AnRIBUTWN OF ATTEMPTEO PERSONAL CHAMQE TO TRAININO 



KMuraup 


NCuA 


NCCD 


AAA A 

(CM 


NJC 






y AT 

a 5. 






AnMliateJuijgn 


N/A 


N/A 


N/A 


3 60% 


TrUJudiitt 




N/A 


1 100% 


74% 73% 


PrivitBAttormyt 


3 S0% 


23 79% 


N/A 


1 100% 


fVoMCuton 


68 80% 


1 50% 


N/A 


N/A 




1 S0% 


57 83% 


N/A 


N/A 


Court Administraton 


N/A 


N/A 


14 52% 


3 60% 


Court Clerici 


N/A 


N/A 


6 50% 


N/A 


Othen 


9 64% 


4 100% 


9 43% 


2 100% 


Total 


81 75% 


85 82% 


30 49% 


83 72% 



AAJE 


UA 


AJC/A6A 


NITA 




N % 


i i 


i 1 


7 88% 


11 79% 


12 55% 


N/A 


16 70% 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


9 82% 


N/A 


N/A 


N/A 


8 89% 


N/A 


N/A 


N/A 


2 100% 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 


N/A 




N/A 


N/A 


3 100% 


23 72% 


11 79% 


12 55% 


22 88% 



30 



34 1 



TABLE 27 

AHRIBUTION OF AnEMPTED ORGANIZATIONAL CHANGE TO TRAINING 



Role Group 


NCDA 


NCCD 


ICM 


NJC 


AAJE 


IJA 


AJC/ABA 


NITA 




N % 


N % 


N % 


N % 


N % 


N % 


N % 


N % 


Appellate Judges 


N/A 


N/A 


N/A 


1 20% 


2 25% 


7 50% 


10 45% 


N/A 


Trial Judges 




N/A 


1 100% 


47 46% 


7 30% 


N/A 


N/A 


N/A 


Private Attorneys 


1 17% 


11 38% 


N/A 


1 100% 


N/A 


N/A 


N/A 


2 18% 


Prosecutors 


33 39% 


2 100% 


N/A 


N/A 


N/A 


N/A 


N/A 

li/ n 


0 JO/0 


Public Defenders 


1 50% 


20 29% 


N/A 


N/A 


N/A 


N/A 


N/A 


2 100% 


Court Adnftinistrators 


N/A 


N/A 


17 63% 


2 40% 


N/A 


N/A 


N/A 


N/A 


Court Clerks 


N/A 


N/A 


8 67% 


N/A 


N/A 


N/A 


N/A 


N/A 


Others 


7 50% 


1 25% 


8 38% 


1 50% 




N/A 


N/A 


2 67% 


Total 


42 39% 


34 33% 


34 56% 


52 45% 


9 28% 


7 50% 


10 45% 


9 36% 
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TABLE 28A 

AREAS OF ATTEMPTED PERSONAL CHANGE - IJA 



Appellate Judges 

Areas N 

1. General philosophical change 1 
in way view profession 

2. General administrative/ 2 
management change 

3. Appellate procedures 1 

4. Writing ability 4 

5. Delegation of responsibility 1 

6. Emphasis on efficiency 1 

7. Noncategorizable 1 
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TABLE 288 

AREAS OF ATTEMPTED ORQANIZATIONAL CHANGE - 



IJA 



Appdtttt JudgM 





fiamn 




1. 


Gen«r«l procsdurai change 


1 


2. 


General administrative/ 


2 




management change 






a. compiiter usage 


1 




b. office filing systems 


1 


3. 


General courtroom-related 


1 




change 




4. 


Relationship with peers/ 


1 




colleagues 
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TABLE 29A 

AREAS OF ATTEMPTED PERSONAL CHANGE - AJC/ABA 



Appellate Judges 

Areas Jjl 

1. General philosophical change 1 
in way view profession 

2. General administrative/ 2 
management change 

3. Appellate procedures 1 

4. Writing ability 4 

5. Delegation of responsibility 1 

6. Emphasis on efficiency 1 

7. Noncategorizable 2 
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TABLE 29B 

AREAS OF ATTEMPTED ORGANIZATIONAL CHANGE - AJC/ABA 



Appellate Judges 



Areas JN 

1. General administrative/ 1 
management change 

a. computer usage 1 

b. docketing 1 

2. Presettlement/pretrial 4 
conferences 

3. General organizational 1 
changes 

4. Noncategorizabie 2 
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TABLE 30A 

AREAS OF ATTEMPTED PERSONAL CHANGE - NJC 



Appellate Judges 

Areas N 

1 . General personal 2 

2. Noncategorizable 1 

Private Attorneys 

Areas S 

1. Noncategorizable 1 

Court Administrators 

Areas ^ 

1. Caseload management 1 

2. Emphasis of efficiency 1 

3. Noncategorizable 1 

Others 

Areas _N 

1. Appellate procedures 2 



Trial Judges 
Areas N^ 

1. Self-confidence 1 

2. Awareness of alternate 1 
approaches to problems 

3. Capacity to solve role- 1 
related problems 

4. General philosophical 3 
change in way view 
profession 

5. General knowledge 1 
change 

a. constitutional law 1 

b. probate law 1 

c. search and seizure 1 

6. General administrative/ 4 
management change 

a. docketing 1 

b. case calendaring 2 

7. General courtroom- 7 
related change 



a. 


use of evidence 


5 


b. 


jury instructions 


5 


c/ 


jury trial behavior 


2 


d. 


filing/use of 


3 




motions 




e. 


plea bargaining 


1 


f. 


presettlement/ 


1 




pretrial conferences 




g. 


sentencing 


6 



8. General personal change 9 

9. Approach to research 1 

10. General change in 1 
communication/inter- 
personal relationships 

a. with public/people 1 
in general 

b. with defendants 2 

1 1 . Emphasis on efficiency 1 

12. Emphasis on training 1 

13. Prioritization of cases 1 

14. Noncategorizable 10 




ll--125g 



O 



TABLE SOB 

AREAS OF ATTEMPTED ORGANIZATIONAL CHANGE - NJC 



Appellate Judges 
Areas N 

Jury selection 1 



Areas 



Private Attorneys 



N 



General organizational 1 
change 



Court Administrators 
Areas 



N 



General administrative/ 1 
management change 
General courtroom- 1 
related change 



Others 



Mreas 



Docketing 



1 



Areas 



Trial Judges 



1. General procedural 
change 

2. Domestic relations cases 

3. General administrative/ 
management changes 

a. docketing 

b. legal research 
indexing 

c. case calendaring 

4. General courtroom- 
related change 

a, jury instructions 
jury selection 

c. jury trial behavior 

d. filing of motions 

e. presettiement/ 
pretrial conferences 

f. sentencing 

5. General organizational 
change 

6. Relationship with peers/ 
colleagues 

7. Emphasis on justice 

8. Emphasis on efficiency 

9. Emphasis on training 

10. Emphasis on post-trial 
phase 

1 1 . Noncategorizable 



2 
5 

6 
1 

1 
3 

2 
1 
1 
2 
1 

1 

5 

1 

1 
1 
2 
1 



f»-l25h . 



TABLE 31A 

AREAS OF ATTEMPTED PERSONAL CHANGE - AAJE 



Appelate Judges 



Areas 



N 



Trial Judges 



Areas 



N 



1, 



2. 
3. 
4. 



General courtroom* 
related change 
a. appellate pro- 
cedures 
General personal change 
Writing ability 
Noncategorizable 



1 
3 
1 



1. General philosophical 
change in way view 
profession 

2. Knowiedge*criminal 
law 

3. Knowledge-procedural 
law 

4. Courtroom-related 
change 

a. pretrial/pre- 
settlement 
conferences 

b. sentencing 

5. General personal change 

6. Writing ability 

7. Communication/inter- 
personal relationships 

a. with public/people 
in general 

b. sensitivity to others 

8. Noncategorizable 
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TABLE 31B 

AREAS OF ATTEMPTED ORGANIZATIONAL CHANGE - AAJE 



Appellate Judges 



Trial Judges 



Areas 



N 



Areas 



N 



General organizational 1 
change 

Emphasis on post-trial 1 



1. 
2. 



Self-confidence 1 
Courtroom-related change 
a. use of evidence 1 



1 



1 



b. presettlement/pre- 1 
trial conferences 

3. General organizational 2 
change 

4. Emphasis on training 1 

5. Noncategorizable 1 
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TABLE 32A 

AREAS OF ATTEMPTED PERSONAL CHANGE - ICM 



Trial Judges 



Areas N 

1. General administrative/ 1 
management change 

Court Clerks 

Areas N_ 

1. General administrative/ 2 
management change 

a. docketing 1 

2. Genera! personal change 1 

3. Emphasis on training 1 

4. Noncategorizable 1 



Others 

Areas N 

1. General administrative/ 2 
management ch^nge 

2. General personal change 2 

3. Approach to research 1 

4. General change in 1 
communication/inter- 
personal relationships 

a. delegation of re- 1 
sponsibility 

5. Emphasis on saving time 2 



Court A dministrators 

Areas ^ 



1. Awareness of issuiss 

2. Capacity to understand 
role-related problems 

3. General philosophical 
change in way view 
profession 

4. General knowledge 
change 

5. General administrative/ 
management change 

a. caseload manage- 
ment 

b. office filing systems 

c. te:.hno!ogical 
changes 

6. Courtroom- related 
pretrial conferences 

7. General change in 
communication/inter- 
personal relationships 
a. with judges 

8. Emphasis on financial 
aspects 

9. Noncategorizable 
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TABLE 32B 

AREAS OF ATTEMPTED ORGANIZATIONAL CHANGE 



-ICM 



Trial Judf M ^ 

Arats N 

General administrative/ 1 
managen>ent change 

Court Clerks 

General philosophical 1 
change in way view 
profession 

General administrative/ 1 
management change 

a. computer usage 2 

b. office filing system 1 
Courtcoom-related 
changes 

a. jury selection 1 

b. filing motions 1 



Court Administrators 
Areas N 

1. Use of resources 1 

2. General knowledge 1 
change 

3. General administrative/ 4 
management change 

a. docketing 1 

b. office filing systems 1 

c. technological 1 
changes 



4. 


d. case calendaring 


1 


General organizational 


1 




change 




5. 


Investigation 


1 


6. 


Relationship with judges 1 


7. 


Enriphasis on financial 


1 




aspects 




8. 


Noncategori^able 


3 



Others 

Areas N 

General administrative/ 2 
management changes 

a. caseload manage- 2 
ment 

b. office filing system 1 
Courtroom-related 
changes 

a. presettlement/ 1 

pretrial procedures 
General organizational 2 
change 
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TABLE 33A 

AREAS OF ATTEMPTED PERSONAL CHANGE - NCDA 





Private Attorneys 






Area 


N 


1. 


General courtroom- 


1 




related change 




2. 


Case preparation* 


1 




investigation 




3. 


Noncategorlzable 


1 




Public Defenders 






Area 




!• 


General personal 


1 




change 






Others 






Area 


IL 


1. 


General knowledge 


1 




change 






a. fraud 


1 


2. 


General personal change 


1 


3. 


Case preparation- 


3 




investigation 




4. 


Writing ability 


1 


5. 


Communication with 


1 




public/people in general 




6. 


Noncategorlzable 


1 



Prosecutors 

Areas N 

1. Capacity to solve role- 1 
related problems 

2. General knowledge 1 
change 

a. constitutional law 1 

b. domestic relations 2 

c. organized crime 1 

d. search and seizure 1 

3. General administrative/ 2 
management change 

a. office filing system 2 

4. General courtroom- 1 1 
related change 



a. 


cross-examination/ 


2 




direct examination 




b. 


use of evidence 


2 


c. 


jury selection 


2 


d. 


jury trial behavior 


4 


3. 


presettlement/pre- 


2 




trial conferences 




f. 


use of technology 


1 


g- 


handling of 


4 




witnesses 





5. General personal change 5 

6. General case preparation 6 
change 

a. case organization 3 

b. collecting evidence 1 

c. investigation 3 

7. Gerieral change in com- 3 
munication/interpersonal 
relationships 

a. with peers/ 1 
colleagues 

b. rfith other law/ 1 
court agencies 

8. Noncategorizable 6 
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TABLE 33B 

AREAS OF ATTEMPTED ORGANIZATIONAL CHANGE - NCDA 



Prrvate attorneys 


Arofli 


N 


neiaiionsnip witn other 


1 


law/iegar agencies 




riiDiic L/eT6nci6rs 




Areas 




diipiiadid on iraimny 


1 


Other 




Area< 


N. 


Org&nJzed crime cases 


1 


Office filing systems 


1 


General organization 


1 


change 




investigation 


1 


General change in 


1 


communication/inter- 




personal relationships 




Emphasis on training 


1 


Noncategorizable 


1 



Prosecutors 

Areas N 

1. Civil law cases 1 

2. Domestic relations cases 2 

3. Organized crime cases 1 

4. General administrative/ 4 
management changes 

5. Investigation 1 

6. Relationship with other 2 
law/legal agencies 

7. Emphasis on training 6 

8. Noncategorizable 2 
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TABLE 34A 

AREAS OF ATTEMPTED PERSONAL CHANGE - NCCD 



Privata attorneys 

Araa> J« 

1 . Capacity to u nderstand 1 
rola-relatad problems 

2. Change related to par- 
ticular types of cases 

a* criminal 1 

b. narcotics 1 

3. General courtroom- 3 
related change 

a, cross-examination/ 4 
direct exanriination 

b. jury instructions 1 
c jury selection 1 

d. presettiemenVpre- 1 
trial conferences 

e. handling of 1 
v^itn esses 

4. General personal change 2 

5. General case preparation 1 
change 

a. collecttr)g evidence 1 

b. forensics 1 

6. General change in com- 1 
munication/interpersonal 
relationships 

7. Emphasis on training 1 

8. Noncategorizable 2 

Prosecutors 
Areas 

1. Emphasis on efficiency 1 



Public Defenders 

Areas 

1. Awareness of issues 1 

2. Self-confidence 1 

3. Capacity to understand 1 
role-related problems 

4. Philosophical change in 2 
way view profession 

5. Personal skills 1 

6. Typesof cases- criminal 1 
law 

7. General courtroom- 10 
related change 



a. 


cross-examination/ 
direct examination 


9 


b. 


use of evidence 


2 


c. 


jury selection 


4 


d. 


jury trial behavior 


1 


e. 


opening/closing 


5 


f. 


handling of 
witneises % 


1 



8. General personal change 4 

9. General case preparation 3 
change 

a. approach to 3 
research 

b. case preparation 1 
notebook 

c. witness preparation 1 

d. writirig ability 1 

10. General change in com- 1 
munications/interper 
sonal relationships 

1 1 . Noncategorizable 4 
response 



Others 

Areas ^ 

1. Self-confidence 1 

2. General courtroom- 1 
related change 

a. use of evidence 1 

b. jury selection 1 
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TABLE 34B 

AREAS OF ATTEMPTED ORGANIZATIONAL CHANGE - NCCD 



Prtvate attorneys 

Areas N 

1. General administrative/ 1 
management change 

2. Courtroom-related 
change 

a. jury selection 1 

b. filing/use of 1 
motions 

3. General organizational 2 
change 

4. General case preparation 1 
change 

5. Relationship with peers/ 1 
colleagues 

6. Emphasis on efficiency 1 



7, Emphasis on training 1 

8. Noncategorizabte 2 



Prosecutors 

Areas N 

1. Relationship with 1 
peers/colleagues 

2. Emphasis on financial 1 
aspects 





Psjhiic riAfanflar* 






Areas 




1 . 


Awareness of issues 


1 


2. 


Office filing systems 


1 


o. 


General courtroom- 


1 




related change 






a. jury selection 


1 




b. opening/closing 


1 




argumenxs 






c. handling witnesses 


1 


4. 


General organizational 


3 




change 




5. 


Investigation 


1 


6. 


Communication/inter- 






personal relationships 






a. with peers/ 


2 




colleagues 






b. delegation of 


1 




responsibility 




7. 


Emphasis on training 


5 




Others 






Areas 


N_ 


1. 


General change in com- 


1 




munication/interpersonal 






relationships 
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TABLE 35A 

AREAS OF ATTEMPTED PERSONAL CHANGE - NITA 



Private Attorneys 
AroM N 

1. General courtroom- 1 
related change 

a. cross-examination/ 2 
direct examination 

2. General case preparatibn 3 
change 

a. case preparation 1 
notebook 

3. RbSstionship wilh peers 1 
4« Emphasis on training 1 



Prosecutors 

Areas 

1. General change in 1 

personal skills 
Z General courtroom- 2 

related change 

a. cross-examination/ 3 
direct examination 

3. Emphasis on post-trial 1 
phase 

4. Noncategorizable 1 



Public Defenders 
Areas N 

1. Case preparation 1 
notebook 

2. Noncategorizabie 1 

Others 

Areas 

1. General courtroom- 2 
related change 

2. General case prepara- 1 
tion change 
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TABLE 3SB 

AREAS OF ATTEMPTED ORGANIZATIONAL CHANGE - NITA 



Private Attomeys 

Emphasis on training 1 
Emphasis on post-trial 1 
phase 



Prosacutore 



Areas 



N 



General case preparation 1 
change 

Emphasis on training 2 



Public Defenders 



Apms 



N 



1. Emphasis on training 1 

2. Emphasis on post-trial 1 
phase 



Others 



Areas 



N 



1. General organizational 1 
change 

2. General case preparation 1 
changa 
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CONCLUSIONS AND RECOMMENDATIONS 
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III. CONCLUSIONS AND RECOMMENDATIONS 



This chapter sets forth the conclMsions reached by the study team after analyzing 
the data collected during the course of this project and reccmmendations which have 
been fonnulated to correct deficiencies and/or strengthen the Courts Training Program. 

These matters will be discussed in five separate sections dealing with training im- 
pact, the training process, cost effectiveness, LEAA national training policy, and future 
evaluations. 



A. TRAINING IMPACT 

Based on the findings of this evaluation, training is noi the major impetus for 
either organizational change within the criminal justice system or personal change 
among those practitioners within the court system. Experience on the job and system 
changes caused by various circumstances emerge as the major causal factors of change. 

Nevertheless, the data indicate that training was c'ted by nearly 20 percent of 
the respondents in all role groups in the 12 court systen^s examined as a generator of 
personal and/or organizational change. Therefore, training's contribution to the total 
climate which produces positive change cannot be dismisseii. 

It is the view of the evaluators that measuring the impact of training per se is 
extremely difficult in light of the various constraints mentioned previously and because 
there is a nat';rdl intertwining and cumulative impacting of such related factors as 
experience on the job, educational background, training, advice of colleagues, reading, 
etc. Despite this, there are sufficient data available to conclude that training is a posi- 
tive and important influence on the criminal justice s/stem and on its practicing 
members 

In examining the impact of training received 2t CT? institutes as compared with 
training received elsewhere, only in two role groups — appellate judges and court ad- 
ministrators — are higher degrees of change tr ;ed o CTP training. Among trial judges, 
prosecutors, defenders and private attorneys ;.nere was little difference in the degree of 
change among the CTP participants and the comparisons, all of whom had like 
amounts of training from nther CTP, state and/or private sources (see Tables 4 and 
4A). On the other hand, comparison group respondents among appellate judges and 
court administrators experienced substantially less training than did the CTP partici- 
pants who tied personal/professional changes to their CTP training. 

Among the eight CTP institutes, those which appear to be most effective in per- 
suading trainees to attempt personal changt"^ in their professional performance are 
NITA, NCCDLPD, IJA, NCDA, NJC, and AAJE. Least effective in this respect are ICM 
and AJC (ABA), according to the data collected. As for encouraging organizational 
change, none of the eight institutes appears to have enjoyed much success, bur fCM has 
been i.iore effective ths.n the others. 

Each of the training institutes will be discussed in the following section. 
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B. THE TRAINING 
PROCESS 



In examining the data gathered from those who have attended CTP institutes, sev- 
eral important overall conclusions emerged: 

• Participants want training. They believe in the principle of continuing 
education, and they feel that the legal system is ever-changing, thereby 
requiring them to increase their knowledge, improve their skills, etc. 
They attend training voluntarily and enthusiastically. 

■ Most sender organizations (e.g., courts, district attorney's offices, oub- 
lie defender's oft* es, etc.) strongly encourage the principle of continu- 
ing education and support attendance at training programs in a variety 
of ways (e.g., financial support, transfer of assignments, rearrangement 
of schedules, etc.). 

■ 77> *^ main strengths of the CTP institutes appear to be in meeting the 
get teral needs of the respective role groups and in providing a setting in 
which individuals can exchange ideas and discuss problems informally 
(often outside the classroom setting) with their peers from other states. 

■ Training, generally, may suffer from shortcomings in needs assessment, 
balance of teaching methodologies (too much lecture), instructor orien- 
tation, and in attempts to meet individual needs. 

■ There is an implied support for training programs which are about a 
week in length. The two, three and four week courses a/e often charac- 
terized as being too long (because of workloads, vacation conflicts 
and family commitments, concentration limits), and the two and three 
day workshops are often seen as being too brief 

■ CTP institutes need to pay more attention to the manner in which they 
mix trainees, especially as to role groups and levels of experience. 

Conclusions and recommendations pertaining to each of the eight CTP institutes 
follow. 

INSTITUTE OF JUDICIAL ADMINISTRATION 

The Appellate Judges Seminars of the I J A emerge from the analysis of data as per- 
hajM the best of the training programs in all respects except cost per participant. 

Tried and tested over a period of 22 years, the seminar program maintains a fixed 
format and program content that appear to have won favor with the more than 800 
judges who have participated. The Appellate Judges Seminars ar?* widely recc jnized 
and respected, as iire the faculty who conduct them. Unlike other training programs 
which at times seem to be more concerned about attracting bodies than reaching target 
audiences, the Appellate Judges Seminars have held the line on attendance limits in 
order ^reserve the atmosphere and format which have been so well received. 
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Go ids and Objectives 
Of The Institution 

The goals of the Appellate Judges Seminar Program have been consistent and ad- 
hered to for more than 20 years. 

Needs Identification 

The methodology pursued by IJA to identify the needs of appellate judges and 
develop the curriculum for the seminars appears to have alleviated the problem identi- 
fied in a previous evaluation, that curriculum content was lodged essentially with one 
individual. The only poteuilal weakness in the present process is the risk of having 
faculty continually recommend that their courses be repeated, without change, not 
necessarily because of "student"needs but because of the faculty member's familiari- 
ty with the course. 



Program Assessment 

The methods currently employed to ascertain the value and impact of the semi- 
nars are of assistar :ib in reflecting the need for changes in faculty and/or curriculum. 
While only the previous outside evaluation of 1975 produced data on how the seminars 
have altered the professional conduct of the appellate ,*Jdges, the fact that "graduates" 
continually recommend the seminars to their colleagues indicates their perception of 
the program. Data collected in this evaluation have confirmed IJA's general effective- 
ness in influencing personal and organizational change. 



Other Institution 
Activities 

We find no evidence of undue competition between the Appellate Judges Semi- 
nars of IJA and those of the ABA in Cnicago. Generally, they are designed for some- 
what different audiences (new appellate judges versus experienced appellate judges), 
and they are presented in different time frames to meet varying needs (e.g., two week 
residential program as compared with four-day sessions). 



Program and 

Related Costs 

The Institute of Judicial Administration is currently contributing between 
$26,000 and $31,000 a year to the Appellate Judges Seminar Program for the travel 
and housing expenses of the participating judges. Many judges are able to obtain travel 
and subsistet^ce funds from their jurisdictions or court training programs, but the dif- 
ferences are marie up by the IJA in the belief that without sjch assistance, appellate 
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judges will not attend the residentia! training programs. We believe that is a question- 
able policy. Given the importance of the seminars to the career development of appel- 
late judges and the fact that many states now provide reasonable salaries for appellate 
judges, we believe that in only the few cases of demonstrated financial need should the 
IJA provide travel and subsistence funds to supplement whatever is provided by state 
sources. If the IJA were gradually relieved of this cost burden, it could assume even 
greater financial responsibility for the Appellate Judges Seminars. 

Recommendation 1: The Institute of Jtidicial Administration should amend its policy 
regarding the provision of travel and housing financial assistance and provide such 
aid only in cases of demonstrated financial need. 

If that recommendation were adopted, IJA would be able to reallocate most of 
the funds now earmarked for travel and housing costs of participants. Such a realloca- 
tion would enable LEAA to further reduce its financial participation in the Appellate 
Judge-? Seminar Program and eventually withdraw completely so that IJA could stand 
with complete independence as the sponsor of the program. I JA's current time deposits 
should prove adequate to support the funding transition, if necessary. 

Recommendation 2: In conjunction with IJA 's adoption of the first recommendation, 
LEAA should further reduce its funding support in FY 1980 and should aim for 
complete withdrawal of financial support by FY 1982. 

APPELLATE JUDGES' CONFERENCE (ABA) 

Data gathered during the course of this evaluation project raise serious questions 
about the value of the Appellate Judges' Conference as it currently exists. The major 
points of concern are: 

■ Twenty three percent of the participants have no desire to return to 
future conferences; 

■ A relatively low percent of the participants have been persuaded to 
mal<e changes in their performance of responsibilities as a result of 
training; 

■ l\/lany respondents gave poor ratings to the AJC programs; 

■ Instructors tend to rate aspects of training design higher than do 
participants, especially as to considering the needs of the individual, 
clarity of learning objectives, the degree to which training objectives are 
demonstrable, the opportunities to practice what is taught, and the 
degree of useful feedback participants get from instructors and peers; 
and 

■ Instructors say that programs are not modified on the basis of feedbacl< 
and systematic assessment. 



Goals and 

Objectives 

The project staff expresses a strong sense of direction: to become a center of con- 
tinuing appellate education and to nneet the ongoing needs of the appellate courts, 
thereby helping to solve the problems of the courts. Although this mission was clearly 
articulated in interviews with the staff, it is not reflected in the written goals of the 
project^ which are stated as exposing members of the appellate courts to ideas, laws, 
administrative techniques, and the like. In other words, the exposure is the means and 
the end. This apparent gulf between the written and spoken objectives of the project 
raised two questions for the study team: do the participants have an understanding of 
'^e expectations of the training; and do their objectives coincide with or complement 
those of the project. Presumably, some common expectations would enhance the im- 
pact of training. Inquiry of the participants indicated that this is, indeed, a deficiency. 



Needs Identification 

Instilling a sense of identity or "ownership" of the project in the participants is 
an abiding concern of the staff. The many processes, however informal, used to deter- 
mine the audience's needs and concerns are vehicles for giving them a voice in the proj* 
ect's development. The seminar evaluation sheets are another such vehicle. 

Because of the informality of the current processes, the study team did not see a 
clear link between the participant's input and refinements in the project's operations, 
as later borne out by the ssjrvey respondents. 

Recommendation 3: A more formal, systematic approach to assessing the needs of the 
audience and its satisfaction with the program would be beneficial to the program 
development process and should be attempted. 

The criteria used to identify and select faculty seem to assure a consistently high 
lev?l of knowledge. If there is any weakness in the faculty, it may be ^he scant repre- 
sentation by members of other than legal disciplines, a point emphasized by respon- 
dents during the field surveys. Considering the emphasis on administrative procedures 
and technological advancements, in addition to substantive law, presumably partici- 
pants could benefit from exposure to members of business, management, and social 
science disciplines. 

A related point is the level of minority representation on the faculty. Included on 
the list of typical faculty, provided by the project, was one woman and one member of 
a minority group. 

Program A ssessm en t 

The staff's expression of a need for outside evaluation {as presented in Chapter II) 
appears to stem from its concern to develop a meaningful program for the participants 



and to achieve practical results. To this end, feedback from participants should assist 
curriculum and program development. 

The current system for project self-assessment focuses only on the appellate jud- 
ges' seminars. The evaluation forms are best described as "happy sheets/' designed to 
assess satisfaction with the format, selection of topics, and physical environment rather 
than to appraise results and learning. The project bases appraisal of its benefits on iso- 
lated reports of participants. This, of course, is a critical source of assessment. How- 
ever, lacking empirical evidence, assessments of widespread benefits or impact are 
basically speculative. Data resulting from this study indicates a low level of impact 



Other Institutional 
Activities 

The project staff is engaged in several outside activities which appear to be very 
beneficial. Most notable is the staff's assistance to and participation in state-adminis- 
tered appellate education programs. This seems to be a positive means of increasing 
outreach. A potential weakness in this activity, however, is that it is not actively mar- 
keted or advertised. This may stem from the limited time that the staff has to devote 
to such efforts. 

Related tc the in-state activity is the input provided to other national judicial edu- 
cation programs. This coordination helps to enhance the consistency of training pro- 
vided to all types of court personnel. 



Program and Related 
Costs 

The staff expressed a desire to explore alternative funding sources so that depen- 
dence on LEAA may be reduced. The study team feels that the project's decision to 
begin charging tuition is a positive step toward this end. 



Management 

The program director expressed a concern that the project is understaffed. It is 
the impression of the study team that the seminar coordination activities per se app --ar 
to be well man?jged. Considering the format of the seminars, many of these activities 
are handled roudnely, with little variation from year to year. 

The fact thist many policies, fiscal procedures, and personnel practices are man- 
aged through the ABA's apparatus certainly contributes to their strength. However, 
as a result of conversations with the staff, it appears to the study team that there is not 
a clear delineation of responsibility and authority among the staff for maximizing effi- 
ciency. In this regard, the program director expressed the concern that an inordinate 



amount of his time is used for attending to routine, day-to-day managerial details. Ac- 
cording to the staff, this results in the program director having little time for other con- 
ference activities and may affect the project's ability to meet its contractual obliga- 
tions. 

Recommendaiion 4: LEAA should increase its monitoring and ongoing evaluation of 
the Appellate Judges' Conference so that it can determine within the next year 
whether the program can be strengthened or whether funding should be termi- 
nated. 

NATIONAL JUDICIAL COLLEGE 

- The data generated as a result of this evaluation indicate that the National Judicial 
College ranks high as an effective training institute, has a well designed training pro- 
gram, and 13 effective in providing a mechanism for exchanges of views among peers, 

NJC has pioneered a number of ideas which have now been replicated by state ju- 
dicial programs. The format of train i ig, novel approaches to sentencing, orientatio:t for 
new judges, emphasis on court admmistratior. and other such features are npw in- 
cluded in state judicial training curriculums. 

NJC gives its participants an intangible feeling of identity as a judge. Most partici- 
pants regard it as a hallmark experience. 



Goats and Objectives 
Of The CoUege 

The College, as evidenced by interviews with its staff and review of its various bro- 
chures, exhibits a strong sense of mission. Its goal of improving the judiciary is based 
on a definite viewpoint what the courts need, a viewpoint which is bolstered 
through continuing contacts with myriad alumni, faculty, and supporters across the 
nation. 

This is manifested in an operational philosophy that emphasizes behavioral rein- 
forcements. For example, the efficient management of time and the service orientation 
exercised by the staff during training are expected to be mirrored by participants in 
their courts. In effect, the College is strong on imaging, or modelling, the behavior it 
wishes to foster in the judiciary. 



Needs Identification 

Judging from the processes depicted by the core staff, the College makes 
thorough attempts to prov de relevant programs to its constituents. Topic areas seem 
to be related to real needs, which are assessed through (1) the Board of DirectorSr (2) 
advisory comrritt'^9s, (3) soundings in the field by the Dean a id other staff members, 
(4) discussion during social events, (5) feedback from participantii. (6) the use of 
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videotapes and peer reactions to mock trials, and (7) field trips to prisons to explore 
the effects of sentencing, and the like. 

In gentral, these approaches seem to be augmented by the College's utilizing the 
"ben sources" for programmatic inputs and refinements, which are obtained, coordi- 
nateo, and implemented by an exceptionatly strong staff based on its qualifications, 
experience, and enthusiasm. Such contributions from both outside resources and the 
competent staff seem to be the result of aggressive and effective outreach and recruit- 
ment. 

Relatedly, College officials described an effective affirmative action program for 
part-time faculty and discussion leaders. Nevertheless, as mentioned in Chapter II, 
minority representation on the core staff (as opposed to faculty) of 36 is zero. This 
raises a question concerning the College's ability to function as a truly national college 
(they recruit some positions nationwide) and to serve adequately those whom the 
courts serve. 

Recommendation 5: NIC should take appropriate and immediate action to ensure 
adequate minority representation on its core staff. 



Program Assessment 

The College constantly updates numerical summaries of its activities and atten- 
dance. For the outsider, these charts provide a useful picture of changing trends, em- 
phases, and clientele over the years. Aside from providing a useful vehicle for public 
relations and marketing, such statistics also allow the staff to assess the degree to which 
the College is fulfilling aspects of its mission (such as the percentage of the potential 
total audience being reached by its programs). 

The College appears to make conscientious attempts to evaluate and monitor pro- 
grams. The statistical and qualitative results of these evaluations are used by the faculty 
and the professional staff for program and curriculum development purposes and by 
the College in general for ma^^keting its services. 

In the view of the study team, however, these evaluations are somewhat unidi- 
mensiona!. They are, in effect, "happy sheets" (albeit good ones) that gauge trainee 
satisfaction rather than systematically assess the degree to which intended learning ob- 
jectives are achieved. (During the site visit, for example, it was mentioned to the study 
team that a particular faculty member from a recent program would not be invited 
back because the participants did not like him. This, of course, may be a valid decision 
when made in consideration of multiple criteria of effectiveness. However, the equating 
of like and learning is questionable.) 

It is not evident to the study team that either measurable performance objectives 
are set for the training sessions or that any systematic evaluation has been done by 
trainers studying actual changes in participant competencies. Further, while the College 
uses some learning theory explicitly (e.g., modelling), in the main it appears that it 
depends on an educational model of course completion raiher than a competency 
model based on performance outcomes. 
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Other College 
Activities 



It was apparent to the study team, through its interviews and review of written 
descriptions, that the College is especially aware of public relations — with outsiders in 
gener: ?s well as with the host University of Nevada/Reno. This awareness seems to 
have at least the following positive effects: first, curriculum materials and publications 
are attentively developed and well marketed; second, people interested in doing leral 
research are encouraged to nail at the College; third, professional disciplines other than 
law are appreciated, as demonstrated by the incidence with which social scientists and 
others are mvited to teach at the College; and fourth, widespread informational con- 
tact is maintained with a significant portion of the College's target audience. 



Program and 

Related Costs 

Due principally to sound fiscal management and increased foundation support, 
the College hss been able to maintain a stable tuition since 1975. 



Management and 
Administration 

As described by the various staff members in individual interviews, the training 
support services — fiscal, personnel, managerial — are strong. Policy manuals provide a 
good delineation of authority, and procedural checklists for quality control are kept 
current. Jobs are similarly delineated, and questions that arise are solved quickly. 

The study team had direct experience with the College's Management Informa- 
tion Systems, which are comprehensive, detailed, and efficient and which allow easy 
access to well-reduced and displayed data. 

The logistics cr choreography of the training sessions are impressive. Tightly 
managed operations are positive in t^at they increase efficiency and the College's com- 
petent image. The trade-off is that tney reduce the ambiguous aspects of the training 
setting, aspects that can be beneficial to learning. 

AMERICAN ACADEMY OF JUDICIAL EDUCATION 

Data collected in the process of this impact evaluation indicate that AAJE is a 
reasonably effective training institute which is well-regarded by those who have at- 
tended its programs. 

Specifically, AAJE participants give high ratings to training design and capability 
of faculty. Exceptionai and widespread praise is given to its legal writing courses. Both 
I^Y and law-trained judges praise AAJE workshops for attempting to integrate judicial 
techniques with matters of ethics, moral philosophy, history and traditions of law, and 
humanities in general. 
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Among the negatives, instructors cite inadequate orientation and training of facul- 
ty, inadequate administrative procedures, high turnover among AAJE staff, and the ab- 
sence of a full-time technical director or leader at all training programs. 

The Academy programs reflect a desire to serve diverse needs of judges. The pro- 
grams are designed for ease of replication, and enrollments in the national conferences 
are consistently high. Our major recommendaiions to the Academy relate primarily to 
its major goals and programs and to questions of management within the Academy 
office. 

Some of our recommendations are offered with the realization that budget and 
personnel constraints would limit a complete implementation of suggestions; but they 
do point to major areas that we believe warrant some change. 



Goals and Objectives 
Of Institution 

Recommendation 6: The Academy should reassess its goals and establish priorities for 
planning. 

The multiple goals of the Academy that Involve serving a wide target population 
do not offe- 3 particular focus for either short- or long-range planning. While its diversi- 
ty of goals lb commendable, the numerous programs and activities tend to dilute the 
resources, both personnel and financial. 

Recommendation 7: The Academy should consider limiting its target audience to 
judges of limited jurisdiction, including those who are nan law-trained. 

Along the lines of the first recommendation, the Academy should consider return- 
ing to r'.^ initial goal of developing programs for judges of lii.iited jurisdiction. In the 
pas; few years, programs have been directed toward the judges of general jurisdiction, 
as well as toward appellate judges. The restriction of the audience will result in fewer 
programs, but the staff and budget resources may be reallocated more effectively. One 
exception to this recommendation is the legal writing cour, js for ^/Z judges. Given the 
wide recognition and appreciation of these courses, they should be continued, if not 
expanded. 



Needs Identification 

liie Academy programs at the present time are selected on the basis of past 
experiences of the Academy and judicial education trends, rather than on a systematic 
needs assessment. Program (: .nning should become a routine activity with regular, 
periodic input from the C Triculum Committee. Because of financial limitations', 
programs need to be e/^amined in depth to determine future directions. 

Recommendation 8: A formal needs assessment should be undertaken c ce the goals 
of the institution have been reexamined. 
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The needs assessment should transcend get it^ral observations about judicial educa- 
tion trends and regularly explore the needs of the target audience. The data gathered 
from a needs assessment should be analyzed and used to develop conference topics, in- 
structional objectives and materials, and teaching methodologies. The needs analysis 
would help in projecting future educational directions and in program planning. 

Recommendation 9: Efforts should be made to implement a faculty development pro- 
gram. 

Currently, there is no provision for orienting faculty to the instructional objec- 
tives or teaching methodologies of the Academy. A program should be implemented 
that would allow faculty and the Academy staff to discuss the training materials to- 
gether. It would also provide some type of consistency of teaching among faculty. All 
new faculty members should receive an orientation package, designed exclusively for 
facu>"Y, which describes the course objectives, teaching methodologies, and available 
training materials. In addition, the faculty should be required to submit, well in ad- 
vance, course outlines regarding both content and format of their specific cleisses. Al- 
though lack of finances has prevented implementation of faculty development in the 
past, some activity could be initiated without a large financial investment. Student in- 
terns could be used to prepare an orientation package. Prrconference meetings between 
staff and faculty would help orient new instructors to the Academy goals and pro- 
grams. A staff representative should ba present at all training programs to monitor 
instructors and to manage the event. 



Other Institutional 
Activities 

Recommendation 10: TIte Judicial Education News, the bimonthly ne^vsletter that was 
terminated for financial reasons three years ago, should be resurrected. 

The newsletter was beneficial in publicizing, on a continuing basis, the Academy 
programs and activities and offered articles relating to current trends in judicial educa- 
tion. While \k may be too much of a financial burden to publish the Newfs biweekly, the 
Academy may consider a monthly, or even a quarterly, newsletter staffed primarily by 
prelaw, law, or journalism student interns. This type of periodical would supplement 
the brochures in providing information to potential participants, alumni, and judicial 
educators, and would provide an ongoing communication between the Academy and 
these groups. 

Recommendation 11: The Academy catalogue, which contains policy and program in- 
formation, should be rewritten for distribution. 

The catalogue, writ'^n three years ago, needs tc be u\ dated with current informa- 
tio.. regarding the Board of Directors, fac jlty, staff, programs, and activities. 
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Recommendation 12: The National Videotape Library should be a priority 
the Academy. 



The Library is a major method of program replication with relatively low develop- 
ment costs. Since the videotape method of instruction is one of the innovations of the 
AciiJemv. the staff should focus its energies on this activity. LEAA should also make 
this matter one of its priority concerns. 



Management 

Recommendation 13: The Academy should analyze the reasons for its relatively high 
staff turnover and take steps to correct the situation. 

Among the issues which should be examined are levels of responsibilities salary 
scales, overall personnel policies, etc. ' 

Recommendation 14: The procedures manual should be revised, completed and av- 
proved by the Executive Director as soon as possible to assure fairness and consis- 
tency in personnel matters. 

The manual should contain job descriptions for all staff and procedures for evalu- 
ation, grievances, and discipline areas which are not presently covered by the manual. 

INSTITUTE FOR COURT MANAGEMENT 

Data gathered in ihe process of this evaluation indicate that ICM is filling a train- 
ing voia for court administrative personnel and that among those court administrators 
and othpr support staff who do attribute changes in their performance to training they 
have received, ICM is mentioned frequently. 

While the relevance of ICM training generally received high ratings from both par- 
ticipants and instructors, more individuals were critical of ICM training than of any of 
the other institutes. According to the data, ICM is not effective, overall, in persuading 
participants to make changes in the way they perform their duties. Some participants 
are critical of the relevance of training programs and the match between trainees' needs 
anr- instructor expertise, instructors cite inadequcte orientation and trai i:-ig and insuf- 
ficient appraisals of their performance. Also, many participants do not share training 
materials with others or recommend ICM. 



Goals and Objectives 
C f The Institute 

The Delphi process undertaken by the Institute to determine needs of court ad- 
ministrators IS an indic-Jtion of the Institute's concern for the vitality and resDOt-sive- 
ness of Its goals and objectives. Additionally, the Board of Trustees the Advisory 



Council of Alumni Fellows, and less formalized needs assessment mechanisms shape 
the institute's understanding of its mission. 

It will be important to see what influence the Delphi findings have with regard to 
one of the current objectives of the Institute: to "professionalize" the position of 
court manager, its supporting strategy, and to consider the human element and culture 
of the courts, rather than simply to treat business management techniques. This issue 
app^rs to have created some ambivalence within the Institute with respect to its seif- 
concept. The question is whether or not the Institute should assume the role of change 
agent/advocate with respect to issues such as local court policy making, use of bail 
bondsmen, non-elected clerks, etc. 



Needs Identification 

I CM programming has shown a creative adaptation to the perceived needs of its 
target group over the past several y^rs. indeed, the Institute is able to react on almost 
a month-to-month basis to the needs of its constituents. Its assessment processes ap- 
pear comprehensive and sensitive to changing needs and opportunities. Also, it appears 
that the Institute's programs are meeting needs that are not already being met by 
other sources (states, universities or other institutes). Yet there is the previously stated 
criticism and dissatisfaction of participants and instructors alike. 

In addition to its regular offerings of education and training programs, the Insti- 
tute engages in research, publishes a first-rate journal, and provides numerous kinds 
of assistance to individuals and organizations — all of which contribute to the Insti- 
tute's stature in court training. 

The approach to training employed by the Institute, androgogy, has both strengths 
and limitations. On the one hand, this approach is most palatable to court professionals 
and utilizes their experience and inputs. On the other hand, the approach tends to tack 
specific direction. 

Recommendation 15: The Institute should develop written policies and procedures for 
orienting instructors to aid them in developing courses and presentations that are 
based on measurable outcomes or objectives. 

The Institute's current course on Strengthening the Executive Component of the 
Court may provide an example of this kind of outcome-oriented training. 

Recommendation 16: The Institute should follow through on its plans to coordinate 
scheduling and programming of training events on court management with the 
National Judicial College. 
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Program Assessment 



The Institute appears keenly interested in improving its program evaluation activi- 
ties. The evaluation form for Strengthening the Executive Component of the Court ap- 
pears to be a significant step away from the less extensive type of end-of-session eval- 
uation in that it focuses on specific performance objectives in the affective, cognitive 
and skill domains. Perhaps this evaluation will provide the Institute with insights cnto 
the measurement of the performance skills of former trainees. 



Other Institute Activities 

Apparently, two areas of potential interface will receive consideration within the 
coming year: 

■ Cooperation with colleges and universities which offer courses re/ated 
to court management; and 

m Interface with the National Judicial College in the scheduling of train- 
ing events and topics. 



Management 

The Board of Trustees and the executive staff appear to function well in their re- 
spective roles and in cooperation with each other. 

The Institute's management information system appears adequate for general 
management purposes. The program performance information and program evaluation 
information might be systematized for use by the Board of Trustees and by program 
managers. 

With regard to personnel, the general stability of the staff tends to reflect ade- 
quate personnel policies. While the Institute offers a generous $1,000 per year for 
tuition and expenses for staff members, and sends staff to conferences and workshops, 
it should also consider an in-house staff development component to enhance its staff's 
performance skills as they relate to program initiatives. 

The fiscal management procedures of the Institute appear thorough and adequate 
for both management purposes and program support. 

The Institute's limited use of its library and locator systems is puzzling. If, in fact, 
the core staff is involved in making recommendatio(^$ for current policy and for pro- 
gram development, it would seem that the library, which is catalogued and kept up to 
date, would receive greater use. (ICM says use of the library has increased as part of its 
stepped-up research efforts.) 
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Challenges and Future 
Directions 



The Institute staff appear keenly aware of the ak^rnatives for the future of the 
Institute (one of which is, surprisingly, the "honorable \termination of the Institute"). 
A healthy openness to the future is reflected in the minutes of Board meetings, execu- 
tive memoranda, the current Delphi process, and the {personal interviews with the eval- 
uation team« 

The study team endorses the current restssessment which t^:: institute is under- 
taking, including the future areas of exploration noted in Chapter u. This reassessment 
is particularly appropriate in view of the problem areas which surf aced in the course of 
this evaluation, LEAA should not be excluded from this reassessment process since it 
too must face some difficult questions regarding the future funding of tCM. 

Recommendation 1 7. Both ICM and LEAA should participate in a reassessment oflCM 
to make determinations concerning future directions, processes and funding 
levels for the Institute. 

NATIONAL COLLEGE OF DISTRICT ATTORNEYS 

NCDA emerges from the analysis of data as a reasonably effective training insti- 
tute, but not without some serious deficiencies. 

On the positive side, NCDA is: 

• Effective in persuading participants to make changes in the way they 
perform their duties; and 

■ Effect've as a mechanism for coHegial contacts and for instilling a 
pride of profession in prosecutors. 

On the negative side, NCDA is criticized for: 

■ Poor mixing of trainees as to roles and experience, overly large classes, 
and insufficient instructor contact after class; 

• Inadequate orientation and training of instructors, inadequate program 
modifications based on systematic assessments, and insufficient apprais- 
al of instructors' performance; 

• Too heavy a reliance on lectures; 

• Sexism and insensitMty to minorities, on the part of both staff and 
faculty; and 

• Lax monrforing of attendance of trainees and indiscriminate awarding 
^^'CLE credits. 
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In a more ne tral mode, there are numerous suggestions by participants and fn- 
istructors for more cooperation and understanding between NCDA and NCCDLPD, 
even to the extent of conducting joint courses. 

The site visit to NCDA indicated that the College is a well-organized institution, 
with sound management and program development methods. The College attempts to 
obtain the best available faculty for its courses and provides outlines for teaching 
assignments that explain the objectives of the course. Based on the critical comments 
of instructors, more training of faculty would be beneficial. 

The College also places much effort in marketing and communications, as re- 
flected in its publications, which include brochures, catalogues, and books. 



Needs Identification 

While NCDA has no formal needs assessment process in place, it utilizes its Re- 
search Attorney, Curriculum Committee, course evaluations and nationwide contacts 
to help identify areas and subjects that require training attention. 

NCDA does not really share the target audience and gools of most of the other 
training institutions covered in this study (except for NITA, to some extent), but 
increased cooperation and sharing with them is feasible in areas of faculty develop- 
ment, teaching methodologies, instructional technology (audio-visual equipment and 
techniques, for example), and general program planning. Collaboration in these areas 
could result in the improvement of programs, avoidance of program duplication and 
decreased costs. 

An example of this type of cooperation would be the attempts of the NCDA to 
explore in more detail the teaching techniques used by the National Institute for Trial 
Advocacy in its seminars. Both institutions could collaborate in designing course ou* 
lines and formats that would integrate each institution's experience and goals, with the 
College taking the lead in determining legrrning needs of the district attorneys and 
NlTA focusing on advocacy techniques whidi it has developed. 



Other Institutional Activities 

The NCDA and the National College of Criminal Defense Lawyers and Public De- 
fenders are housed within the same building and presently share some facilities and 
services. While we understand the potential difficulty of undertaking joint activities, 
especially with regard to staff and equipment, we would suggest that greater effort be 
exercised in this direction. For example, one receptionist could be utilized to serve 
both colleges. The NCDA may also wish to reconsider the offer of the NCCDLPD to 
lease computer time on its equipment. The NCDA had initially rejected an offer to do 
so because of differences in accounting methods and in compiling mailing lists, but 
if this arrangement would result in considerable savings, the NCDA may wish to re- 
evaluate the offer even if certain changes in operating procedures are necessary. 
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Student evaluations of courses conducted by both colleges indicate a desire of the 
participants to have a representative from the other school available at the workshops 
to offer another perspective. These two colleges should work more closeiv in determin- 
ing which of their courses may benefit from joint participation. 

Recommendation 18: The National College of District Attorneys should explore addi- 
tional possibilities for sharing facilities, services, and personnel with the National 
College of Criminal Defense, including the possibility of joint training programs. 



Management 

NCDA was the only CTP institute criticized for the insensitivity demonstrated 
by its staff and faculty at regional workshops in regard to sexism and monito.'ities. 
Coupled with the findings stated in Chapter II regarding the composition and charac- 
teristics of NCDA's core faculty and central staff, the institute appears to have several 
related problems which must be addressed. 

Recommendation 19: NCDA should reorganize its core faculty and central staff to pro- 
vide for a more realistic representation of women and minorities. 

At the same time, concrete steps should be taken to educate faculty and those 
staff assigned to training programs as to the proper ways to relate to other individuals. 
Formal sensitivity training provided by professionals may be in order. 

Some participants complained about, and we obser.'ed, the laxity of attendance 
monitoring at NCDA workshops and the attendant indiscriminate awarding of CLE 
credits, even to those who failed to attend most sessions. 

Recommendation 20: NCDA should alter its training program monitoring procedures 
(and perhaps its monitoring personnel) to ensure that trainees are encouraged to 
attend sessions and that repeated absence will in some way be reported back to 
the sender organization or at least not be rewarded with the presentation of 
CLE credits, 

NATIONAL COLLEGE OF CRIMINAL DEFENSE 
(LAWYERS AND PUBLIC DEFENDERS) 

NCCDLPD also emerges from the data analysis as a relatively effective training 
institute. It persuades a high percentage of its trainees to adopt new performance 
methods and techniques, and it is highly commended for reinforcing the role and 
confidence of the public defenders as a professional group. 

The only major criticism of its teaching methods is that lectures are used too 
extensively while participants feel they benefit more from the lesser-used role playing/ 
simulation techniques. 
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As in the case of NCDA, participants at this institute recommend more integra- 
tion and cooperation of and between the two, including joint training programs. 

Goals and Objectives 
of the Institution 

While the stated goals and objectives of the institution have remained constant 
over the years, we share the same doubts that previous outside evaluators have ex- 
pressed concerning the College's priority of improving the quality of representation 
for the indigent defendant. 

There is no evidence that the College has taken any meaningful action or steps to 
con-ect the problems cited by previous evaluators in 1974 and 1976 which have al- 
lowed private attorneys to attend LEAA-funded programs - sometimes with scholar- 
Pliip assistance - even though there was little evidence of their past or future inclina- 
tion to defend the indigent. Although the College now says it notifies the courts in 
which these private attorneys practice that they have received training and are avail- 
able to accept assignments to defend indigents, the College takes no action to see 
if such assignments are made and accepted. Even the application process, whereby 
the private attorney states a previous record and future willingness concerning rep- 
resentation of indigent defendants, has a ring of superficiality to it. 

In fairness to the College, we found instances during the field survey in which 
NCCDLPD participants (private attorneys} did want but could not get court appoint- 
ments to defend indigents. However, we also found too many instances of private 
attorneys, some of whom attended NCCDLPD on LEAA scholarships, who defended 
few indigents prior to training, few if any since training, and virtually none now since 
they have switched to civil practice. 

We believe this matter to be of paramount importance to LEAA, and we will 
have recommendations later in this chapter which bear on the issue. In any event 
however, we believe that LEAA should take the following action: 

Recommendation 21: LEAA should closely monitor the future activities of the Na- 
tional College of Criminal Defense Lawyers and Public Defenders to ensure 
that program attendance is limited to attorneys providing some reasonable 
amount of representation to indigent defendants. 



Needs Identification 

The College has no formal procedures for assessing the training needs of the 
defender community. Instead, it relies on trainee and faculty feedback, plus the 
experience of its own staff. 

Recommendation 22: NCCD should move to develop a more systematic method 
for identifying the needs of the audience it serves. 



LEAA discussed this recommendation with NCCD prior to the publication of this report. It 
appears that the College is already taking steps to address this recommendation. 
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Management 

We have two concerns regarding the faculty of the National College of Criminal 
Defense Lawyers and Public Defenders. First, we believe the composition of the 
faculty, with only eight or nine percent female and four or eight percent black (de- 
pending on which of NCCD's figures are used), is unrepresentative of the public defen- 
der's field and today's society. It cannot be ignored that, at the trnie of our visit, the 
College's staff also lacked sufficient minority representation. 

Second, while some attention has been paid to the preparation of faculty for the 
two-week summer resident institutes, there is little evidence that such preparation is 
accomplished for other programs offered by the College. {This problem was noted in 
the previous outside evaluation, in 1976.) Further, the attorney occupying the po- 
sition of Director of Training has been doing very little in the way of training or in any 
way utilizing his legal background. His main functions have concerned physical arrange- 
ments for the workshops and institutes. 

Recommendation 23: The College should reorganize its core faculty group to assure 
greater representation of qualified women, blacks, and other minority groups. 

Recommendation 24: The College should take steps to assure staff representation 
of racial and ethnic minorities^ in the spirit of equal employment opportunity. 

Recommendation 25: The College should assign the Dean and Associate Dean to 
assume responsibilities formerly assigned to the Director of Training for the 
preparation and training of faculty for all programs^ and such preparation should 
be strengthened. 

Program Assessment 

As indicated previously, there are indications that the College is not vigorously 
pursuing its stated commitment to improve the quality of representation for the 
indigent defendant. Any lawyer who pays full costs may attend the LEAA-funded 
training programs regardless of his/her interest in the indigent defendant. Up to fifty 
percent of the scholarships may be and are awarded to private attorneys who simply 
state that they have in the past, or will in the future, represent indigent defendants. 
Verification of such statements, before and after the training, is not attempted. 

Recommendation 26: LEAA should require the College to become more selective 
in the awarding of scholarships to private attorneys and insist on verification of 
the applicant's record of representing indigent defendants, * 

The College should also be required to conduct a follow-up analysis of such 
scholarship recipients to determine if the courts notified by the College do, in fact, 
appoint these trainees to defend indigents. 

* LEAA discussed the recommendation with NCCD prior to the publication of this report. It ap- 
pears that the College is already taking steps to address this recommendation. 
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Two important issues which surfaced as a result of the two previous outside evalu- 
ations have to do with the matter of private attorneys who do not represent indigent 
defendants and the lack of preparation of faculty members. In neither case is there 
evidence that the College has acted to correct these previously identified - and still 
apparent - deficiencies. Therefore, LEAA's monitoring of ihe College in the future 
should pay particular attention to the actions taken to implement the recommenda- 
tions contained In this report and agreed to by both parties. 



Other Institution 
Activities 

While certain economies have been effected as the result of the sharing of space 
and certain equipment between NCCDLPD and NCDA, additional steps could be taken 
to further reduce costs without undue inconvenience. Such steps could include the 
sharing of a joint receptionist at the entrance of the office suite, the joint purchasing 
of videotape equipment, and the coordination of selected training workshops to share 
costs of lecturers, equipment and space. We also believe that NCDA should give further 
consideration to the sharing of NCCDLPD's computer. 

Recommendation 27: NCCD and NCDA should formally develop plans for the in- 
creased sharing of costs, including those suggested in this report, to take fuller 
advantage of their physical proximity and related interests. 

Program and 

Related Costs 

NCCDLPD's daily cost per student compares favorably with other 'raining institu- 
tions, However, the amount of money being expended annually by the organization 
could be further reduced by either eliminating or sharply curtailing the awarding of 
scholarships to private attorneys, as another means of dealing with the issue of indiqent 
representation. 

While we make no specific recommendation in this regard, we suggest that LEAA 
explore these possibilities. 

NATIONAL INSTITUTE FOR TRIAL ADVOCACY 

K„T^^^ °" ^^^^ gathered and analyzed during the course of this evaluation 
NITA compares favorably with the two other trainers of advocates but with some 
notable differences. 

On the positive side, NITA boasts sound teaching techniques which are widely 
recognized, praised and replicated. Participants are highly laudatory of the role play- 
mg/simulation/videotaplng process. Perhaps as a result, NITA is the most successful 
CTP institute m persuading its participants to change personal practices on the basis of 
training experience. 
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On the negative side, having attended one NITA training progrann, participants 
feel there is not much more to be offered, and 40 percent do not wish to return. In 
addition: 

• There is some criticism of the wa/ in which trainees are mixed as to 
roles and experience; 

• Participants do not reflect the s^me level of devotion or allegiance to 
NITA as do other CTP participants to their respective institutes; and 

« There is evidence of serious management problems within NITA. 

Finally, there is the observation that, based on the statistics and data available 
to us, relatively few public employees (prosecutors, defenders, etc.) attend NITA 
programs. Rather, there seems to be a preponderance of private attorneys, many in 
civil practice, whose attendance is supported financially by their firms, if this is the 
case, to what extent if any should federal funds be financing such training? Should it 
be limited, for example, to scholarships for prosecutors and pubiic defenders? 



Goah and Objectives 
of the Institution 

(It must be noted here that since the study team's visit to the NITA offices in 
Chapel Hill, N.C. a new director has be^n designated and the NITA headquarters 
shifted to South Bend, Indiana. Comments contained in this report relate to the situ- 
atibn as we found it in Chapel Hill). 

Within the framework of limited but powerful goals, NITA has adopted a set of 
teaching techniques. Those are carefully controlled and are seemingly maintained at a 
high level of purity. There is standardization of curriculum, materials and teaching 
format. In addition, there is ongoing monitoring as a result of management observa- 
tion, peer discussions, student comments, and professional evaluations. It is claimed 
that despite the growth in their regional programs and the development of in-house 
and bar association workshops, there is little sag in the overall quality of the work 
that they are doing. 

The programs have been tested since their inception in 1971. In general, the value 
of the programs to lawyers engaged in both civil and criminal trial advocacy, as well as 
law school teachers, seems to be evidenced by the fact that overall enrollment does not 
seem to be a problem: law school teachers are competing to enter the special work- 
shops designed for them, the number of regional programs are growing, and law firms 
are sending their junior members to regional and national sessions. 

The overarching goal of the National Institute for Trial Advocacy is to contri- 
bute to the development of an adequately trained, professionally responsible trial bar, 
sufficient to serve the needs of justice in the United States. It strives to achieve this by 
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training trial lawyers, and especially young lawysrs, in advocacy; by developing meth- 
ods and techniques for teaching the skills of the effective, professional trial advocate; 
and by oncouraging the teaching and learning of these skills - to train teachers for ser- 
vice in law schools and continuing legal education programs. They also attempt to 
stimulate the creation of courses and programs in trial advocacy, and to aid in their 
development. These goals and objectives seem to have been consistent th. oughout the 
life of the Institute since 1971. 



Needs Identification 

Program development is largely vested in the director who is advised by perma- 
nent faculty and the board of trustees. He seems to be responsive to inputs from these 
sources and, in addition, makes use of the evaluations of the program made by the 
participants, as well as the follow-up studies that have been made by the professional 
evaluator whom the Institute employs. The director calls his permanent faculty mem- 
bers together for an annual planning meeting, at which time the direction of the Insti- 
tute's programs is debated. 



Program Assessment 

NITA is one of the few institutions currently under study which attempts to as- 
sess the impact of its programs on participants' careers. For this purpose it employs 
the impact studies generated by Dr. Russell Burris, a professional evaluator. to fine 
tune curriculum. In general. Dr. Burris' evaluations reported that ten months following 
enrollment in NITA's programs, the participants tended to see continuing improvement 
in their skills as trial advocates. References, such as judges, prosecutors and senior 
partners, for whom the participants work, tended to confirm the self-evaluations. 



Other Institution 
Activities 

We find little evidence of undue competition between the National Institute for 
Trial Advocacy and other programs that are engaged in the teaching of the subject of 
trial advocacy such as the National College of Criminal Defense Lawyers and Public 
Defenders and the National College of District Attorneys. While there is some unavoid- 
able overlapping of teaching area, there is a tendency on the part of the former to 
emphasize pretrial skills, while the major emphasis of the NITA program deals with 
trial dynamics. In addition, the national colleges have programs designed to teach 
management skills, areas that are not covered in the NITA curriculum. The major 
weaknesses in this area are the failure of the three institutions to share information 
about training materials and techniques or, for that matter, to utilize each other's 
materials. 
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Ptogrmm and 

Related Costs 



The annual bv^dgei: is approximately $650,000. The largest contributions to that 
budget are tuition fees which are now $1 ,250 for national and regional programs. Since 
NITA is basically for young lawyers tn the first five years of their careers, this substan- 
tial sum can frequently bo a sacrifice. Hence, it has devoted the grants from LEAA 
(this year their grant is $121,000) toward helping to defray the costs of ti tion for 
needy participants by awarding scholarship funds. The institute is developing addi- 
tional funding through gifts, grants, and the sale of teaching materials. They have de* 
velopad an outreach program to alumni and to law firms in an effort to attract more 
funding. Contributors in such fund raising tend to come from the civil bar and would 
inevitably steer the Institute's programs toward civil cases. In addition, it would seem 
that further cuts in funding by LEAA would tend to reduce the numbers of young 
district attorneys and public defenders from enterinc this program. 

Recommendaiiott 28: In view of the negative impact tftat a further reduction of fund- 
ing would have on NITA *s ability to grant scholarships to public attorneys en- 
gaged in criminal practi^, LEAA should attempt to maintain its current level of 
funding support for such scholarships. 



Management 

The findings section of this report (Chapter II) noted a number of management 
deficiencies. While no one of these is of a critical magnitude, their combined existence 
would indicate that some tightening of management and administrative procedures 
would be beneficial. One serious weakness is the difficulty in obtaining current infor- 
mation regarding expenditures and budget status. We have previously noted that some 
of the problems we encountered on site in seeking to obtain financial information were 
due to the inexperience of the new bookkeeper/office manager, the tag time of the 
tocal data processing service in generating monthly reports, and the location of some 
financial records in Minneapolis. The bookkeeper will gain experience and familiarity 
with the NITA operation, and we were told that the local data processing service is 
improving to the point that monthly financial status reports are being produced 
within the acceptable period of two weeks following completion of the reporting 
period. The renrv^ining step that should be taken by NITA is to centralize its financial 
records at the NITA office rather than having them spread over two locations. 

A second serious weakness was the absence of an up-to-date file on the faculty, 
There were folders scattered throughout the file, some of which were no longer active. 
Files of some current faculty members did not exist. We described in Chapter II how 
the names of eight black faculty members were recited to us, although there was no 
record of their existence in the faculty files. On the whole, and based on the data we 



EKLC 



III -2:i 



were able to collect, the faculty did not seem particularly impressive in terms of law 
experience, teaching experience and publications. But this impression may be unjusti- 
fied since it is the result of scanning credentials which are in need of updating and 
expansion. 

This confusion over NITA records extended to the field survey phase where 
lists supplied by NITA, supposedly of participants practicing/residing in 12 specific 
gec^raphic locations, proved most unreliable at times. Similar confusion existed with 
respect to NITA-suppiied lists of its instructors and governing board members. 

Recommendation 29: NITA should unify the geographic location of its administra- 
tion and establish a more efficient method of records management. If the new 
central offices of NITA are to be established in South Bend, then all records 
should b'd assembled there, not dispersed in Chapel Hill or Minneapolis. The 
records management system is especially crucial to NITA's file of faculty mem- 
bers and past trainees, plus its financial records. 

Although the size of the NITA central staff was relatively small at the time of 
our visit to Chapel Hill, personnel matters seemed to be conducted on an informal ad 
hoc basis. Certainly should the size of the staff be increased, and perhaps even at its 
present size, these procedures should be formalized through the development of a 
personnel handbook and procedures manual. 



C COST EFFECTIVENESS 

Developing cost effectiveness statistics for the eight CTP institutes is both diffi- 
cult and somewhat precarious* The difficulty is traced to the origin of data concerning 
program costs. These data are supplied by the training institutes. Each has its own 
method for arriving at those costs, and there are indications that accuracy is far from 
consistent. The precariousness concerns the output measures. The impact measures 
used in this case are the percentages of participants of each institute who claim to 
have attempted personal and organizational changes in their professional pursuits as 
a result of training. The effectiveness rating is a somewhat subjective assessm'snt of 
each institute which is based on a combination of management and training strengths 
and weaknesses which the data reflect. 

Finally, there is a fallacy in attempting to compare one institute with another 
solely on the basis of cost effectiveness. Each of the institutes offers some inherent 
qualities and serves some useful purposes which are somewhat intangible, cannot be 
easily measured, and should not be quickly discarded solely on the basis of cost effec- 
tiveness calculations. 

Cost impact is measured in terms of the ratio between dollars expended and the 
amount of change or impact registered The basic cost measure used is the average cost 
per student per day of each institute (See Chapter II). 

When the basic cost measure is applied to the impact measure of personal chanqe 
attempted as the result of training, the rank order of the eight institutes is: 
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■ National College of District Attorneys 

■ National College of Criminal Defense 

• American Academy of Judicial Education 

■ Appellate Judges' Conference 

■ National Judicial College 

■ National Institute for Trial Advocacy 

■ Institute for Court Management 

■ Institute of Judicial Administration 

When ihe basic cost measure is applied to the impact measure of organizaiional 
change attempted as the result of training, the rank order of the eight institutes is: 

■ Appellate Judges' Conference 

■ Wational College of District Attorneys 

■ National College of Criminal Defense 

■ National Judicial College 

■ American Academy of Judicial Education 

■ Institute for Court Management 

■ National Institute for Trial Advocacy 

■ Institute of Judicial Administration 

When the basic cost measure is applied to the overall impact measure of personal 
and organizational change attempted as the result of training, the rank order of the 
eight institutes is: 

■ NDtional College of District Attorneys 

■ National College of Criminal Defense 

■ Appellate Judges' Conference 

■ American Academy of Judicial Education 
" National Judicial College 

■ National Institute for Trial Advocacy 

■ Institute for Court Management 

■ Institute of Judicial Administration 

The danger of this type of analysis is best demonstrated by the case of IJA, In 
terms of impact, it ranks first among all of the CTP institutes. Yet it ranks last in cost 
impact because the program pays all travel and subsistence costs of the participants. 
Removing those costs from the calculations would improve IJA's standing* in the 
rank order only slightly. It is an expensive program when costs are compared with the 
small number of trainees to which each class is deliberately limited. Is the high impact 
(and high quality) of the Appellate Judges Seminars worth the high price? Conversely, 
is tha Appellate Judges' Conference of greater value because its low basic cost measure 
offsets its low impact ranking? 

In an effort to provide yet another approach to the question of cost effectiveness, 
a program effectiveness rating was applied to each of the eight CTP institutes. The 
rating, on a scale of one (low) to ten (high), included considerations of training impact, 
training effectiveness, and management influences ~ virtually every facet of the data 
collected during visits to the institutes, observation of their training programs, and the 
questioning of participants, supervisors, comparisons, instructors and board members. 
The one consideration omitted was program costs. 
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The program effectiveness ratings result in the following rank order: 

• Institute of Judicial Administration 

■ National Judicial College 

■ American Academy of Judicial Education 

■ National College of District Attorneys 

• National College of Criminal Defense 

■ National Institute for Trial Advocacy 
" Institute for Court Management 

■ Appellate Judges' Conference 

Hcv/r^-.o when the basic cost measure is applied to the program effectiveness 
rating, th3 cost effective rank order of institutes that results is: 

■ National College of District Anorneys 

■ . National College of Criminal Defense 

■ American Academy of Judicial Education 

■ Appellate Judges' Conference 

• National Judicial College 

• National Institute for Trial Advocacy 

• Institute for Court Management 

• Institute of Judicial Administration 

Note that the institutes which rank 1-2-3 in program effectiveness rank 8-5-3 in 
cost effectiveness. 

There are three other measures of effectiveness upon which rankings could be 
based. For example, if sole consideration is given to the effectiveness of the eight in- 
stitutes in encouraging their trainees to attempt changes in the way they perform their 
work, the rank order is: 

■ National Institute for Trial Advocacy 

• National College of Criminal Defense 

• Institute of Judicial Administration 

» National College of District Attorneys 

• National Judicial College and the American Academy 

of Judicial Education 

■ Appellate Judges' Conference 

■ Institute for Court Management 

If sole consideration is given to the effectiveness of the eight institutes in en- 
couraging their trainees to attempt organizational changes, the rank order is: 

■ Institute for Court Management 

■ Institute of Judicial Administration 

■ National Judicial College and Appellate Judges' 

Conference 

• National College of District Attorneys 

• National Institute for Trial Advocacy 
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■ National College of Criminal Defense 

■ American Academy of Judicial Education 



And finally, if sole consideration is given to data which show which of the OTP 
institutes are most frequently cited by those participants who do credit training for 
changes they have undertaken, the rank order is: 

■ National Institute for Trial Advocacy 

■ Institute for Court Management 

• American Academy of Judicial Education 

■ Institute of Judicial Administration 

• National College of Criminal Defense 

• National Judicial College 

■ National College of District Attorneys 

■ Appellate Judges' Conference 

Regardless of how the cost impact or cost effectiveness statistics are interpreted 
and used, there are four institutes which should be monitored by LEAA relative to 
program costs. These are AJC, NCCDLPD, NCDA and AAJE. Regardless of the way in 
which they calculate their program costs and display their overhead expenses,there are 
reasons to believe that an excessive percentage of the annual LEAA grants are being 
used for general operating expenses of the institute rather than direct program pur- 
poses. This question should be examined more closely by LEAA. 



D. LEAA/NATIONAL 

TRAINING POLICY 

As a result of the data collected during the course of this evaluation, a number of 
issues have surfaced which can be addressed only by LEAA. Our conclusions and 
recommendations in these matters are discussed below. 

We have found numerous examples where LEAA appears to be competing with 
itself in the field of courts training* 

The assertion by the Dean of the NCCDLPD that the National Institute of Law 
Enforcement and Criminal Justice has funded seven defender management workshops 
through another organization, without consultation or discussion with the College, is 
one example. In addition to the questioned wisdom of one unit of LEAA funding a 
program in competition with a program funded by another unit of LEAA, there is 
an even more critical issue as to whether LEAA has considered the experience of the 
NCCDLPD with defender management workshops. If there is a low attendance re- 
sponse to the one management workshop which has been presented annually by 
NCCDLPD, what rationale is there for LEAA to now finance seven management work- 
shops each year. It is difficult to justify recommended decreases in spending by 
NCCDLPD on the one hand, then see LEAA expend even greater amounts of money 
for programs which have not been well received and for which the need has not been 
proven. Further, given the absence of consultation with NCCDLPD in this matter. 
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we must question the degree of coordination that exists within LEAA and the willing- 
ness to share and utilize information that will be generated by evaluations such as this. 
The same kind of competition, or duplication, is found in other national and state 
programs, many of which receive some degree of financial support from LEAA. 

Recommendation 30: LEAA should develop an inventory of all courts training pro- 
grams it funds, regardless of pnmary supporting division or office, for the pur- 
pose of coordinating the allocation of such funds. The inventory should be ex- 
panded eventually to determine the number and nature of training programs 
being conducted by public agencies and private organizations in each state. The 
SPA'S and RPU's could be called upon to assist in this undertaking. 

During the field surveys in the 12 case study sites nationally, we found little 
evidence of State Planning Agency and Regional Planning Unit support of courts 
training. Few monies are being allocated by SPA's, or recommended by RPU's, for 
these purposes. In fact, the level of awareness of the need appeared to be relatively low 
among those agencies. 

Recommendation 31: LEAA should consider ways to make SPA 's and RFUs more 
aware of the importance of courts training and to encourage increased local 
funding support, where warranted. 

The study team visited three training institutions which have interrelationships as 
to the trainees they reach and the subject matter they teach. These are the National 
Institute for Trial Advocacy, the National College of Criminal Defense Lawyers and 
Public Defenders, and the National College of District Attorneys. In all three cases, 
we find it disturbing and potentially wasteful that there is no sharing, and often little 
knowledge, of the training materials and techniques developed by each. If one of the 
anticipated benefits of the courts training program was the replication of training 
materials and approaches, the program has failed in that respect. 

Specifically, the National College of Criminal Defense Lawyers and Public De- 
fenders has virtually no knowledge of the instructional materials and techniques that 
have been developed by NITA, nor does NITA have knowledge of the College's. 
Insofar as the College is concerned, we detected no interest in utilizing any materials 
provided by other institutions, especially NITA, which is clearly recognized as the 
College's only competitor. 

Recommendation 32: LEAA should take steps, through budgetary action and program 
regulations, to require funded training institutes to share training materials and 
exchange ideas on effective teaching techniques. This exchange could be effected 
through periodic meetings of institute directors, through seminars offered by 
LEAA in which institute representatives would participate, and through inter- 
institute newsletters which focus on innovations in programs, administrative pro- 
cedures, planning, equipment, curriculum, alternate funding sources, etc. There 
are a number of ways in which this problem can be attacked, but action should 
be taken to eliminate this potential for waste of money and duplication of efforts. 
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A growing problem which became evident during the course of the field sur- 
veys is the limited availability of out-of-state travel funds for those who wish to attend 
courts training programs. In response, participants are turning to private organizations 
and institutions in or near their states or to the increasing number of in-state courts 
training programs, regardless of their limitations. 

Recommendation 33: LEAA should discuss with the CTP institutes the wisdom of 
placing a greater emphasis on regional training programs as an answer to increase 
ing travel costs and limitations on out-of-state travel In addition, if regional 
training programs would focus on local practices and situations, it would offset 
complaints that national training programs are too general and not relevant to 
the participant's jurisdiction. 

Because of funding limitations, organizational structure, and other factors, 
LEAA monitors the CTP institutes to a very limited extent. Most of the institutes have 
never or only rarely been visited by a representative of LEAA. Few if any of their 
training programs have been observed first-hand by LEAA representatives. Most 
"monitoring" is conducted by means of telephone conversations, mail exchange, and 
periodic meetings in Washington. 

The monitoring of the CTP by LEAA is clearly inadequate under present con- 
ditions. 

Recommendation 34: LEAA should move to increase its monitoring of CTP institutes 
by attending/observing at least two training programs of each institute annually 
and by visiting each institute at least once a year. The monitoring activities should 
tie into an ongoing system of impact evaluation, which is discussed in the next 
section. 



E. EVALUATION METHODOLOGY 

As pointed out earlier in this report, one of the constraints faced in this evalu- 
ation was the fact that an evaluation process was not built into the Courts Training 
Project from the outset. While periodic evaluations of this nature have value, they are 
not as productive, nor as timely, as an ongoing evaluation which measures impact, 
accompanied by a monitoring component which alerts management promptly if po- 
tential problems develop. 

We have recommended that LEAA increase its monitoring of the CTP. We also 
suggest that LEAA implement a system of ongoing evaluation of the CTP institutes. 

We have developed an evaluation manual which LEAA personnel may use as a 
guide in planning and conducting the type of impact evaluation recommended (See 
Appendix C). The manual closely parallels the techniques utilized in conducting this 
evaluation, so its contents are practical and proven. 
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While evaluation is a valuable management tool, it is also an expensive one which 
requires ample funds and staff to accomplish. We recognize this may prove to be an 
obstacle to LEAA. Nevertheless, if LEAA is to know what is being accomplished with 
the money being spent, then evaluations must be conducted — and certainly on a more 
frequent basis than has been the case in the history of the Courts Training Project. 



111-30 



SUMMARY OF RECOMMENDATIONS 



IJA 



1. The Institute of Judicial Administration should amend its policy regarding the 
provision of travel and housing financial assistance and provide such aid only in 
cases of demonstrated financial need. 

2. In conjunction with IJA's adoption of the first recommendation, LEAA should 
further reduce its funding support in FY 1980 and should aim for complete 
withdrawal of financial support by FY 1 982. 



AJC 

3. A more formal, systematic approach to assessing the needs of the audience and 
Its satisfaction with the program would be beneficial to the program develop- 
ment process (of the Appellate Judges' Conference) and should be attempted. 

4. LEAA should increase its monitoring and ongoing evaluation of the Appellate 
Judges' Conference so that it can determine within the next year whether the 
program can be strengthened or whether funding should be terminated. 



NJC 



i. NJC should take appropriate and immediate action to ensure adequate minority 
representation on its core staff. 



AAJE 



6. The Academy should reassess its goals and establish priorities for planning. 

7. The Academy should consider limiting its target audience to judges of limited 
jurisdiction, including those who are nonlaw-trained. 

8. A format needs assessment should be undertaken once the goals of the institution 
have been reexamined. 

9. Efforts should be made to implement a faculty development program. 
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10. The Judicial Education News, the bimonthly newsletter that was terminated for 
financial reasons three years ago, should be resurrected. 

11. The Academy catalogue, which contains policy and program information, should 
be rewritten for distribution. 

12. The National Videotape Library should be a priority concern of the Academy. 

13. The Academy should analyze the reasons for its relatively high staff turnover and 
take steps to correct the situation. 

14. The procedures manual should be revised, completed, and approved by the 
Executive Director as soon as possible to assure fairness and consistency in per- 
sonnel matters. 



ICM 

15. The Institute should develop written policies and procedures for orienting instruc- 
tors to aid them in developing courses and presentations that are based on meas- 
urable outcomes or objectives. 

16. The Institute should follow through on its plan to coordinate scheduling and pro- 
gramming of training events on court management with the National Judicial 
College. 

17. Both ICM and LEAA should participate in a reassessment of ICM to make deter- 
minations concerning future directions, processes and funding levels for the 
Institute. 



NCDA 

18. The National College of District Attorneys should explore additional possibilities 
for sharing facilities, services, and personnel with the National College of Criminal 
Defense, including the possibility of joint training programs. 

19. NCDA should reorganize its core faculty and central staff to provide for a more 
realistic representation of women and minorities. 

20. NCDA should alter its training program monitoring procedures (and perhaps its 
monitoring personnel) to ensure that trainees are encouraged to attend sessions 
and that repeated absence will in some way be reported back to the sender organ- 
ization or at least not rewarded with the presentation of CLE credits. 
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NCCD 



21. LEAA should closely monitor the future activities of the National College of 
Criminal Defense to ensure that program attendance is limited to attorneys 
providmg some reasonable amount of representation to indigent defendants. 

22. NCCD should move to develop a more systematic method for identifying the 
needsof the audience it serves. 

23. The College should reorganize its core faculty group to assure greater represen- 
tation of qualified women, blacks, and other minority groups. 

24. The College should take steps to assure staff repcesentation of racial and ethnic 
minorities, in the spirit of equal employment opportunity. 

25. The College should assign the Dean and Associate Dean to assume responsibilities 

ZniTJX ^^u '° °* ^'■^'"'"9 the preparation and training of 

faculty for all programs, and such preparation should be strengthened. 

26. LEAA should require the College to become more selective in the awarding of 
scholarships to private attorneys and insist on verification of the applicant's 
record of representing indigent defendants. 

27. NCCD and NCDA should formally develop plans for the increased sharing of 
costs, including those suggested in this report, to take fuller advantage of their 
physical proximity and related interests. 



NITA 



Mi-!^A^ "^Sative impact that a further reduction of funding would have on 

rrnl I ca I I scholarships to public attorneys engaged in criminal prac- 

tice, LEAA should attempt to maintain its current level of funding support for 
such scholarships. hki^hiui 

mI3^.«>""'* ""'^ the geographic location of its administration and establish a 
more efficient method of records management. 



LEAA 



h!frHio'«°"i'^ ^" inventory of all courts training programs it funds 

regardless of primary supporting division or office, for the purpose of coordi- 
nating the allocation of such funds. K H «i ^.uurai 
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31. LEAA should consider ways to make SPAs and RPUs more aware of the impor- 
tance of courts training and to encourage increased local funding support, where 
warranted. 

32. LEAA should take steps, through budgetary action and program regulations, to 
require funded training institutes to share training materials and exchange ideas 
on effective teaching techniques. 

33. LEAA should discuss with the CTP institutes the wisdom of placing a greater 
emphasis on regional training programs as an answer to increasing travel costs 
and limitations on out-of-state travel. 

34. LEAA should move to increase its monitoring of CTP institutes by attending/ 
observing at least two training programs of each institute annually and by visiting 
each institute at least once a year. 
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